I have a folder of ~200 000 files with a total size around 300 GB that I need to transfer over an SSH connection to a remote host. I'm looking for a way to have the local files deleted after it's been verified that they've been successfully copied to the remote host, but I can't figure out how to do it. rsync --remove-source-files
seems unsafe if connection breaks according to other posts (here). How can it be done safely?
3 Answers
Have a shell script:
Note that it makes some assumptions, which I've documented below.
#!/bin/sh
# first, upload the directory structure
# be REALLY sure that it made it intact!
while true
do
# try to upload it:
rsync $flags --filter="+ */" --filter="- *" $source $destination
# it uploaded fine? cool, break the loop (else try again):
[ "$?" -eq 0 ] && break
done
# now the files
for file in $(find $source -not -type d)
do
# again, be REALLY sure they copied okay!
while true
do
# try to upload it:
rsync $flags $file $destination/$file
# it uploaded fine? cool, break the innermost loop (else try again):
[ "$?" -eq 0 ] && break
done
# delete the local copy of the file:
rm $file
done
This script assumes the following:
- That
$source
,$destination
, and$flags
were either set as environment variables, or replaced in the script with the actual source, destination, and any otherrsync
flags you want used. (Don't replace$?
or$file
.) - That
$source
is be a relative path. - That
rsync
will exit with a nonzero status if for any reason the specified$file
was not copied 100% successfully. - That you've tested it on less important data, because I haven't.
The rsync
--filter
parameters were copied from a blog post by some guy called Phil. I hope that's okay with him. :)
-
Cool, thanks. If I understand the script, it will create a new connection for each file - won't that slow down the transfer substantially? Though, I guess if it gets the job done it's worth it anyway :)– MorganSep 19, 2013 at 5:59
-
You're absolutely right—it will use a new instance of
rsync
for each file, preventing connection reuse. I can't think of another way to get the files deleted as soon as they're done uploading, though, except maybe usingstrace
or something, and deleting files asrsync
closes them? Sep 19, 2013 at 11:13 -
You might combine my script with Kent's solution. Instead of making a list of all the files and copying them one by one, make a list of subdirectories with a few files in each (you can be the judge of how many to upload at a time). Again, test it out with less important things, first; I'm not sure how
rsync foo/bar $server:foo/bar
behaves with an existingfoo/bar
directory on the server—it might copy the localfoo/bar
to inside the remote one, giving youfoo/bar/bar
and breaking your directory structure! Sep 19, 2013 at 11:23 -
You can work around the latter problem by assuming there are no duplicates in the directory list and deleting the empty folders (
ssh $server rm -r $destination/$file
) before you callrsync
. That does introduce more overhead, though. Grrrrr… (I hope you have passwordlessssh
login set up, too. Needless to say, this will require logging into the server many, many times.) Sep 19, 2013 at 11:27
May be you can use sshfs. sshfs lets you "mount" a remote directory via ssh (actually performing commands other ssh to mimick a file system). You can then use mv
to move your files.
You don't need to be superuser, since it's a userspace filesystem.
-
1
-
Isn't
sshfs
somewhat unreliable? I'm guessing Morgan wantsrsync
for its verification (else a simplescp -r $folder $host:$folder
would suffice), and if that's the case, I wouldn't want to usesshfs
. (The alternative would be usingrsync
oversshfs
, defeating the purpose of usingsshfs
in the first place.) Sep 18, 2013 at 14:25 -
1If the source and destination are both on the same filesystem, the link to the file is modified. If the file is on a different filesystem, the file is first copied and the original file has its link removed. If the connection breaks before the file is copied, you may potentially have a partial file, but the original would be left alone. Sep 18, 2013 at 14:25
-
@Blacklight Shining: depends on what you call unreliable (we use it on a daily basis). The point is that if you let the connection inactive for some time, one endpoint (either client or server) may decide to close the connection. If you perform a sshfs / mv (or cp) / fuserumount, there won't be any trouble, since the connection is always used. If you perform an sshfs in the morning and expect the connection to be up say an hour later, you have to be sure your sshd and possibly the client are configured so that the connection is maintained up. Sep 18, 2013 at 14:36
-
1Hi Morgan. UtahJarhead answered: if the connection breaks for any reason during file transfer, you'll have a partially transfered file on the receiving side, and your file intact on the sending side (since file on the sending side is only deleted after the transfer). Sep 18, 2013 at 16:01
You could split the files up into blocks of ten for example, and then
rsync (source files) destination
And, if it ran successfully, follow it with
rsync --remove-source-files (source files) destination
The added bandwidth of a second rsync command is negligible.
-
No need to run
rsync
again; it'd just waste time checking to make sure that the files are all the same—which it just did, in the first invocation. Afterrsync
, you can just dorm -r $source_files
. Sep 19, 2013 at 11:17 -
Doh! Good point. I got caught up in the
rsync
euphoria. It -may- provide a slightly larger sense of security, in that the second rsync checks the work of the first one... but, I think that would be minimal at best.– KentSep 19, 2013 at 23:00
rsync $arguments && rm -r $folder
? That should invokersync
, and only run therm
ifrsync
exits successfully.rsync
one subdir at a time, removing it on successful upload? (Do you need the solution to be automated, or will you be more or less at your computer the entire time?)rsync --remove-source-files
but more stable.