Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am logged in via ssh to server1, while on server1 I want to get files located in a git repository on server2 and save them on server1 to do something with them (a script to save to a database)

The git repository is quite a deep structure. I can't just clone the repository because I run out of space. So of course I want to loop through all the files on Server2, and with each one move it to server1, do my function to save to the database, and then delete the file.

The part that I am having trouble with is recursively looping the files to download over ssh.

on edit: I am communicating between two redhat servers.

share|improve this question

1 Answer 1

This is not very elegant, but hopefully the idea will get across:

ssh localhost -p4444 \
 "find ~/tmp/  -iname \*png -exec \
    sh -c 'echo mkdir -p moo-diles/\`dirname {}\`\;\
           rsync -e \"ssh -p 4444\" localhost:{} moo-files/{}\;\
           rm -i {}' \; "

(...)

mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/tile; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/tile/veins_2.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/tile/veins_2.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/tile/veins_2.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/menu2.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/menu2.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/menu2.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_2.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_2.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_2.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_2.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_2.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_2.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_up_3.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_up_3.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_up_3.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_3.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_3.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/bar_vertical_3.png
mkdir -p moo-diles//home/jaroslav/tmp/eterm/usr/share/Eterm/pix; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_1.png moo-files//home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_1.png; rm -i /home/jaroslav/tmp/eterm/usr/share/Eterm/pix/button_arrow_down_1.png
mkdir -p moo-diles//home/jaroslav/tmp/transcode; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/transcode/track14.png moo-files//home/jaroslav/tmp/transcode/track14.png; rm -i /home/jaroslav/tmp/transcode/track14.png
mkdir -p moo-diles//home/jaroslav/tmp/transcode; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/transcode/closeup.png moo-files//home/jaroslav/tmp/transcode/closeup.png; rm -i /home/jaroslav/tmp/transcode/closeup.png
mkdir -p moo-diles//home/jaroslav/tmp/transcode; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/transcode/track15.png moo-files//home/jaroslav/tmp/transcode/track15.png; rm -i /home/jaroslav/tmp/transcode/track15.png
mkdir -p moo-diles//home/jaroslav/tmp/transcode; rsync -e ssh -p 4444 localhost:/home/jaroslav/tmp/transcode/track10.png moo-files//home/jaroslav/tmp/transcode/track10.png; rm -i /home/jaroslav/tmp/transcode/track10.png

(...)

What's being done here can be called "meta programing" because for every found file, we simply echo more commands to execute. These commands must be later executed to process each file. You will be wise to put the commands that come in sh -c '<here>' in a script on the server instead. That way you'll avoid losing your sanity while trying to escape things properly and reading a command that is three lines long.

share|improve this answer
    
ok actually that looks pretty good, but I ended up fixing my problem in another way - just mounting an NFS share where I could store as much data as I need. –  user254694 Nov 7 '12 at 20:40
    
Good thinking. In the future, if you don't have root on the boxes you could possibly use sshfs (FUSE). –  Ярослав Рахматуллин Nov 7 '12 at 20:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.