Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

To get rsync cronjob working, I need to first create the dir tree on the remote server before running rsync. The local directories shall mirrored (w/o content) to the remote server. I'm strugglying to do that.

What I need is stg like this :

find /home/local_directories/onedir/* -type d -exec sh -c "ls -dlts {}"  \; | 
    cut -d" " -f10 | uniq | ssh user@server mkdir -p /tmp/{xargs}  

The hard point is how to get the last pipe running ?

EDIT: the rsync command is like this (tried also with -R):

/usr/bin/rsync -e ssh -arz --stats --force /home/local_directories/onedir/ user@server:port:/exitsing/path/to/remote/dir/  

The source tree is a svn hotcopy (/conf, /db, /hooks, /dav, ...)


share|improve this question
Can you add your rsync command along with a sample source directory tree and the desired target directory tree? I don't know if you need to be running mkdir repeatedly on the target host. – Mikel Feb 28 '11 at 0:18
cut -d" " -f10 is probably wrong. Are you trying to get the directory name? Or what it points to if it's a symlink? Or something else? – Mikel Feb 28 '11 at 0:33
Please don't parse ls. – Dennis Williamson Feb 28 '11 at 0:36
Indeed, once we know what -exec sh -c "ls -dlts {}" is for, then we can replace it with find -printf or stat or whatever. – Mikel Feb 28 '11 at 0:41
@Mikel,@Dennis :I agree this is an ugly way to get the sub directories path from /home/local_directories/onedir – hornetbzz Feb 28 '11 at 0:48
up vote 1 down vote accepted

It's not clear why you need to make all those directories using mkdir.

The only directory you should need to create is the destination directory itself, and that's only if it doesn't already exist. rsync will create all the subdirs if you use the -a or -v option.

Something like this should be sufficient:

destdir="/home/local_directories/onedir.bak"  # or wherever it should go
ssh user@server "mkdir \"$destdir\""
rsync -a /home/local_directories/onedir/ user@server:"$destdir"

Or more correctly (only making the directory if it doesn't exist):

ssh user@server 'destdir="'"$destdir"'"; test -d "$destdir" || mkdir -p "$destdir"'
rsync -a /home/local_directories/onedir/ user@server:"$destdir"

But, if you really have to do this, and for some reason the ls -ltds really is required, then you would have to fix your {xargs} bit to get it running.

The most correct and efficient way I can think of to do the last bit is:

find ... |
    ssh user@server 'while IFS=$'"\n"' read -r dir; do test -d "$dir" || mkdir -p "$dir"; done'

The idea behind this is that we only start find and ssh once. We start find and ssh together in a pipeline, with find sending its output to the remote server over ssh.

find produces one line of output per file, then the while read loop on the other side reads one line at a time, and does something with it (in this case, makes the directory).

If you were using xargs, it would be running ssh multiple times, which will make things slower.

Better, but a bit more complicated, is:

find ... -print0 |
    ssh user@server 'while read -r -d "$(printf "\000")" -r dir; do test -d "$dir" || mkdir -p "$dir"; done'

It's better because it handles weird characters in files, including newlines, which are very rare, and a bad idea, but better to handle them just in case.

So as far as I see it, you should just be doing something like:

find /home/local_directories/onedir/ -type d -print0 |
    ssh user@server 'while read -r -d "$(printf "\000")" -r dir; do test -d "$dir" || mkdir -p "$dir"; done'
share|improve this answer
rsync is used many times in a threaded bash script. It fails in some cases when remote dir tree does not exist or may have disappeared for other reasons on the remote server. – hornetbzz Feb 28 '11 at 0:38
OK, so next can you explain what you want ls -dlts | cut -d" " -f10 to do? There are some better ways. – Mikel Feb 28 '11 at 0:44
this is an ugly way to get the sub directories path from /home/local_directories/onedir, actually even worse : find /home/onedir/* -type d -exec sh -c "ls -dltsha {}" \; | cut -d" " -f10 | uniq | sed -e "s/.*onedir//" - Currently strugglying with the find/while method that I did not know – hornetbzz Feb 28 '11 at 0:58
thx, that helped a lot to get it done in a proper way. – hornetbzz Feb 28 '11 at 1:09

I don't quite understand what you're trying to achieve here. I use rsync all the time and have never had to pre-make any directories on the server.

rsync -avP /home/local_directories/onedir/ user@server:/where/to/sync/to/

should just mirror the whole tree under onedir from local to remote.

share|improve this answer
remote server dir tree may change in some cases. – hornetbzz Feb 28 '11 at 0:43
thx to your comment, I also checked the rsync command threads and could notice that it should NOT include the port in user@server:PORT IF the command is keychained via ssh, otherwise it fails saying the remote dir does not exist. – hornetbzz Feb 28 '11 at 6:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .