Questions tagged [rsync]
Rsync is an open source command line tool for Unix and Windows systems that provides fast incremental file transfer to copy files and synchronize directories locally or between remote hosts. It is widely used for mirroring and backup.
1,397
questions
0
votes
0
answers
45
views
Ramdisk speeds on older server bottlenecked when rsyncing files vs dd
Why am I seeing poor copy performance in memory?
Background and details:
While testing a RAID 10 on an older server, I wanted to see whether or not other parts of my system were bottlenecking my ...
- 238
1
vote
1
answer
3k
views
rsync exclude directory and all subdirectories
I'm trying to use rsync to synchronise some of my data but I'm unable to exclude what I want. This is my directory tree:
tst/
├── st1
│ ├── st11
│ │ └── txt.text
│ ├── st12
│ ├── st13
│ ├──...
- 113
0
votes
1
answer
334
views
rsync: chown "/some-destionation-path/.somefile.jpg.pY2TRG" failed: Permission denied (13)
I'm doing rsync -rlptoghv for transferring a folder from XFS to ZFS. Usually, no problems arise. But sometimes, for some files (and I can't find a pattern), rsync fails to sync file permissions and ...
- 11
2
votes
2
answers
349
views
Why Can't I copy a folder to a disk that is larger than my folder?
I am using ubuntu 18.04-4 and want to move a large directory (yocto build directory for my project) from my ~/Desktop to an external drive (ext4 formatted). The external drive is a 512 GB empty drive. ...
1
vote
1
answer
5k
views
rsync failed: Permission denied (13)
I am trying to use "rsync" to copy file to another machine.
This is the command I am trying:
sudo rsync -avztP --delete --recursive --inplace /folder1/folder2/folder3/myfile [email protected]....
- 111
0
votes
2
answers
67
views
Rsync files based on highest alpha within a date
I'd like to rsync a folder which contains files in the format
fixed string<yyyy-mm-dd><version letter>.xml
The version letter is a, b, c, ... and I only want the files with the latest ...
- 101
0
votes
1
answer
535
views
How to get rsync to print stats when interrupted
I ran rsync with the --stats option, because the sync is being done over the metered connection. However, after some time I wanted to get some idea of how much had been transferred and then (...
- 2,714
0
votes
2
answers
899
views
How to rsync that excludes .git/ folder but still keep the .git/config file?
In brief TLDR
To rsync a folder that has .git/ excluded but still keeping the .git/config file, how to do that?
My google search is not helpful so I asked here.
Full details
I want to take snapshot of ...
- 12.3k
0
votes
1
answer
729
views
rsync compare destination issues
Update for Clarity I want to sync two directories - a desktop folder & external hard drive. Additionally, I want to copy all the new/changed files between the source directory (desktop folder) &...
- 1
0
votes
1
answer
596
views
Mac OSX Catalina and Rsync: failed to open logfile (name): Read-only file system
Ok, so I'm usually on stackoverflow, and I'm admittedly a little out of my element when it comes to more sys admin stuff.
I had an rsync script I wrote myself that functioned fine before I updated to ...
- 103
2
votes
0
answers
703
views
How to efficiently distribute large, incrementally updated sqlite database?
What is an efficient way to transfer deltas from a large sqlite database?
Note: this question is NOT about how to copy/distribute a "live" database. I.e. I am not concerned about corrupted ...
- 121
0
votes
1
answer
2k
views
How to copy files from mounted directory?
I want to copy files from my mounted dir to my HOME directory.
My /mnt
/mnt# ls
'01 - Course Overview' '03 - Locating Elements & Navigating Dynamic Web Pages' '05 - Overcoming ...
- 491
2
votes
2
answers
2k
views
backup windows folders with rsync from a linux server
Windows now has a build in openssh server, so we can use rsync to backup folders on windows pcs too.
Problem is, that if I try to
rsync --relative -aqz --numeric-ids --delete-after -e 'ssh' 's2@black:...
- 372
0
votes
0
answers
201
views
How to start a secondary rsync task if and only if a primary rsync task completes successfully?
Background
I have two servers, ServerSource and ServerDestination (running Ubuntu 18.04), connected over the internet.
I'm using rsync (running on ServerDestination) to make a (public) mirror on ...
- 289
0
votes
1
answer
2k
views
How to see the progress of rsync's receiving and building file lists?
I was trying to use rsync to move over a TB of files from one device on my gigabit LAN to the machine I am on and wanted to be able to see the progress of rsync.
from my machine (fedora 31) I started ...
- 1
2
votes
1
answer
2k
views
auto-sync directory on two linux machines over SSH
I have a root folder on two computers (that I run programs on alternately). Let's say:
server0:
-my_dir
--subdir0
--subdir1
server1:
-my_dir
--subdir0
--subdir1
So I have batch ...
- 201
1
vote
1
answer
483
views
rsync FROM EC2 instance
I'm trying to copy data from an EC2 instance to a remote host and am wondering why my rsync command hangs.
I've generated a key-pair using ssh-keygen on the EC2 instance and then added the id_rsa.pub ...
- 111
4
votes
0
answers
814
views
Copying files of Windows partition using rsync gives readlink_stat(...) Input/output error (5)
I want to copy all files of a Windows 10 system partition (NTFS) to another partition (NTFS).
I am using rsync under Ubuntu like this:
sudo rsync -axHAWXS --numeric-ids /media/windows /mnt/windows-...
- 212
1
vote
2
answers
5k
views
How to copy folder structure without files into a new directory on MacOS Catalina?
I've looked this up and tried everything using rsync and involving find ., but keep getting errors. Using rsync I get "rsync error: errors selecting input/output files, dirs (code 3)", and ...
- 11
0
votes
1
answer
2k
views
RSync, Copy all images but not directory structure
I'm trying to copy all images from many subdirectories into one folder without the current folder structure.
Example:
product/v/9/v9315.jpg
product/v/9/v9316.jpg
product/v/9/v9317.jpg
product/v/9/...
- 101
0
votes
1
answer
73
views
Calculate on rsync output with wc -l
When I run an bash script, I want to know how many files were moved with rsync.
I though about doing a calculation like this
$(rsync -avz ./* ./ | wc -l)-4
The minus 4 is because rsync always outputs ...
- 1
1
vote
0
answers
161
views
Effect of option --usermap, but for older rsync
I would like to run a rsync command to copy data from my laptop (Linux Mint) to my NAS (Synology) over SSH. I would like to also use the --usermap option. However, the version of rsync on the NAS is ...
- 312
1
vote
1
answer
386
views
Rsync only when external drive is connected
How can I make rsync happen ONLY if I have my external backup drive connected? When I set a cron job to do this, it worked fine. But, when I disconnected the drive, it went and created a folder /media/...
- 136
2
votes
1
answer
6k
views
How to log the output from an rsync backup which runs from a script using crontab?
I would like to log all the output that is generated by rsync; the output that I see when I run the script myself in terminal. I would prefer this logging to occur from within my script.
Here is my ...
- 23
1
vote
0
answers
18
views
rsync following symlink and deleting only the symlink upon completion
expected scenario
Essentially I want to use rsync as a tool to do automatic queued downloading.
Say we have the following directory structure on remote.
to_be_downloaded
├── symlink1
└── symlink2
...
- 111
1
vote
1
answer
94
views
Why isn't rsync synchronizing this file?
I have two versions of a file, each in a different directory:
$ dir foo{_test,}/bar.js
-rw-r--r-- 1 ikegami pg9999999 53721 Jun 9 15:11 foo/bar.js
-rw-r--r-- 1 ikegami pg9999999 53721 Jun 9 21:12 ...
- 234
0
votes
0
answers
242
views
FreeBSD rsync server: please follow symlinks in target directory
Summary
rsync server on FreeBSD won't follow local symlinks to dirs. It will either error out (can't write file) or clobber the symlink and create an actual directory, which is not the desired ...
- 147
0
votes
1
answer
987
views
Getting error while using Rsync on Cygwin
I am using rsync, to transfer data to HPC cluster, using Cygwin for windows 7. I type the following command to transfer a folder from my computer to the HPC remote server.
rsync -rzv /cygdrive/C/...
- 133
0
votes
1
answer
211
views
Copying Data from a hard drive to another hard drive with different partitioning
I have a System HardDrive - "sda", 64GB big with the following partitioning:
sdb is a second harddrive, 128GB big, on which sda is cloned with rsync. It has only 1 partition, taking the full size of ...
0
votes
1
answer
175
views
Gitlab and Rsync updates servers files incorrectly
I don't know how necessarily say this correctly, as I simply am not sure what is happening.
I've got a website that is being updated using gitlab ci. When attempting to push new code today I received ...
- 131
1
vote
1
answer
1k
views
rsync - exclude files in directories with specific conditions
I am setting up an rsync based backup from our FreeNAS server to a QNAP server.
The structure looks something like this:
Sequence
├── SHOT01
│ ├── comp
│ │ ├── work
│ | │ └── images
│ |...
- 13
1
vote
0
answers
190
views
Why can't rsync hard link certain files on a Mac?
I am running rsync to sync files from Linux to a mounted Mac OS X (HFS+) disk image.
I am getting a lot of errors of the form:
rsync: link "/Volumes/Archive/blahblahblah" => blahblahblah failed: ...
- 2,714
1
vote
2
answers
807
views
Cloning hard drive with rsync and accidently deleted home folder
I want to replace my old SSD hard drive with a new one that has double the size, but somehow fail to understand how rsync is working.
I read this post Copy entire file system hierarchy from one drive ...
- 123
1
vote
0
answers
143
views
Why simple-mtpfs rsync stalls when transferring files?
I am mounting my USB-connected Android phone in my Linux box with the command
simple-mtpfs -o allow_root -o enable-move /mnt/phone
It mounts fine, I can see both my internal and external storages, ...
- 181
3
votes
2
answers
1k
views
Are backed up files from an encrypted /home folder themselves encrypted?
I'm setting up my backup and restoration scripts on my Linux Mint machine to an external hard-drive using rsync.
When I installed the system it was marked the option to encrypt the /home partition.
...
- 65
0
votes
1
answer
2k
views
rsync command with multiple jump servers
I have a server than I can only access via 2 jump servers:
user@mylaptop$ ssh jumpserver1 ssh [email protected] ssh targetserver "bash -c 'hostname;whoami'"
targetserver
user2
...
0
votes
1
answer
703
views
trouble with spaces in environment variable in rsync script on MacOS
I have a Bash script that I plan to run for daily backups (my HOME is /Volumes/Norman Data/me):
#!/bin/bash
# Halt the script on any errors.
set -e
# Remote server where
remote_server="example.com"
...
- 133
0
votes
1
answer
130
views
Why does it say no space available when a lot are avaliable?
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 30G 290M 28G 2% /
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs ...
- 123
2
votes
0
answers
49
views
Copy symbolic link and its target subdir to new dir
I have a dir with a symbolic-linked subdirectory:
greg:/testA> ls
foo -> alice
alice/
I need to copy the link "foo-> alice" (preserve it at destination) AND the subdirectory alice/ to a new ...
- 21
1
vote
1
answer
435
views
How to prevent a corrupted file from being rsynced to remote server and good/non-corrupted copy being on the destination deleted?
I am using this command:
rsync -avzh --delete --progress source folder remote folder
My source folder contains many video *.mp4 files.
I want all the videos to be rsynced to a destination folder in ...
4
votes
2
answers
10k
views
Best way to compare two very large directories in Linux
I need to compare two directories on Linux filesystems on two separate servers in order to identify whether all the files from SERVER1 are present on SERVER2. The total data set is about 4TB of files ...
- 141
2
votes
0
answers
891
views
rsync: send_files failed to open "filename": Invalid argument (22)
I use rsync to copy files from a mounted davfs box.com folder to an ext4 partition. For files containing the symbols [, ", and {, I receive the following error:
rsync: send_files failed to open "...
- 21
1
vote
1
answer
857
views
rsync --remove-source-files that exist on remote
I'm periodically calling rsync --remove-source-files -r source/logs/ dest:logs/ to synchronize log files from a mobile unit with inadequate storage to my reliable server with ample storage.
Sometimes,...
- 326
0
votes
0
answers
358
views
Is there a way to keep two directories (on two servers) in sync with each other, truly?
I have two servers in AWS that we are using to store a software packages codebase that is accessed by the web application/server.
/var/www/vhosts/application-dua/content/
This same folder structure ...
- 125
1
vote
2
answers
64
views
DIfferent behaviours of rsync
According to what I have understood, rsync without the -o and -g options is meant to copy without changing the owner and group of the destination. However, I found out a very strange behaviour of ...
- 125
0
votes
1
answer
525
views
Access drive by label under rsync/cygwin?
I have downloaded rsync for Windows.
It uses cygwin.
My rsync command is this:
rsync.exe -W -a -v --delete --exclude=cache2 --exclude=Cache "/cygdrive/C/Users/user1/" "/cygdrive/F/backup/"
As you ...
- 1,250
0
votes
3
answers
2k
views
Rsync don't create directory at target if if doesn't exist
I have set rsync to backup the root filesystem on a Raspberry Pi to an USB drive every day using /usr/bin/rsync -ahPHAXx --stats --delete-during / /mnt/backup/pi-backup/. However, if the USB disk isn'...
- 3
1
vote
1
answer
5k
views
Terminate SSH Connections That Hang or Become Idle
I have come across some instances where an SSH connection on my Linux OS will become idled, freezing and staying open. Eventually, this causes a fork bomb and the following error messages:
/etc/...
- 133
2
votes
1
answer
591
views
Deal with reserved characters when syncing volumes with different formatting
I have a NAS and I want to copy some directories on it to an exfat formatted external HDD.
Problem is paths / filenames may contain reserved characters for exfat (see https://en.wikipedia.org/wiki/...
- 121
0
votes
0
answers
190
views
rsync - list all options alphabetically?
I'm just starting to get acquainted with rsync... it has hundreds of options, both short (single-dash) and long (double-dash).
The man pages don't list these alphabetically, either under "OPTIONS ...
- 665