1

I have a folder of ~200 000 files with a total size around 300 GB that I need to transfer over an SSH connection to a remote host. I'm looking for a way to have the local files deleted after it's been verified that they've been successfully copied to the remote host, but I can't figure out how to do it. rsync --remove-source-files seems unsafe if connection breaks according to other posts (here). How can it be done safely?

4
  • What about rsync $arguments && rm -r $folder? That should invoke rsync, and only run the rm if rsync exits successfully. Sep 18, 2013 at 14:19
  • I need the files to be removed during the transfer, after each individual file (or a small number of files) has been copied. (The reason for this simply being that I need space on the local drive and that the connection I'm copying over is annoyingly slow.)
    – Morgan
    Sep 18, 2013 at 14:24
  • Might you split the files up into subdirectories, then rsync one subdir at a time, removing it on successful upload? (Do you need the solution to be automated, or will you be more or less at your computer the entire time?) Sep 18, 2013 at 14:30
  • I need to keep the folder structure, so I can't make subfolders. I would prefer it to be automated. Basically rsync --remove-source-files but more stable.
    – Morgan
    Sep 18, 2013 at 15:53

3 Answers 3

1

Have a shell script:
Note that it makes some assumptions, which I've documented below.

#!/bin/sh

# first, upload the directory structure
# be REALLY sure that it made it intact!
while true
do
    # try to upload it:
    rsync $flags --filter="+ */" --filter="- *" $source $destination
    # it uploaded fine? cool, break the loop (else try again):
    [ "$?" -eq 0 ] && break
done

# now the files
for file in $(find $source -not -type d)
do
    # again, be REALLY sure they copied okay!
    while true
    do
        # try to upload it:
        rsync $flags $file $destination/$file
        # it uploaded fine? cool, break the innermost loop (else try again):
        [ "$?" -eq 0 ] && break
    done
    # delete the local copy of the file:
    rm $file
done

This script assumes the following:

  1. That $source, $destination, and $flags were either set as environment variables, or replaced in the script with the actual source, destination, and any other rsync flags you want used. (Don't replace $? or $file.)
  2. That $source is be a relative path.
  3. That rsync will exit with a nonzero status if for any reason the specified $file was not copied 100% successfully.
  4. That you've tested it on less important data, because I haven't.

The rsync --filter parameters were copied from a blog post by some guy called Phil. I hope that's okay with him. :)

4
  • Cool, thanks. If I understand the script, it will create a new connection for each file - won't that slow down the transfer substantially? Though, I guess if it gets the job done it's worth it anyway :)
    – Morgan
    Sep 19, 2013 at 5:59
  • You're absolutely right—it will use a new instance of rsync for each file, preventing connection reuse. I can't think of another way to get the files deleted as soon as they're done uploading, though, except maybe using strace or something, and deleting files as rsync closes them? Sep 19, 2013 at 11:13
  • You might combine my script with Kent's solution. Instead of making a list of all the files and copying them one by one, make a list of subdirectories with a few files in each (you can be the judge of how many to upload at a time). Again, test it out with less important things, first; I'm not sure how rsync foo/bar $server:foo/bar behaves with an existing foo/bar directory on the server—it might copy the local foo/bar to inside the remote one, giving you foo/bar/bar and breaking your directory structure! Sep 19, 2013 at 11:23
  • You can work around the latter problem by assuming there are no duplicates in the directory list and deleting the empty folders (ssh $server rm -r $destination/$file) before you call rsync. That does introduce more overhead, though. Grrrrr… (I hope you have passwordless ssh login set up, too. Needless to say, this will require logging into the server many, many times.) Sep 19, 2013 at 11:27
0

May be you can use sshfs. sshfs lets you "mount" a remote directory via ssh (actually performing commands other ssh to mimick a file system). You can then use mv to move your files.

You don't need to be superuser, since it's a userspace filesystem.

8
  • 1
    How does mv handle broken connections?
    – Morgan
    Sep 18, 2013 at 14:22
  • Isn't sshfs somewhat unreliable? I'm guessing Morgan wants rsync for its verification (else a simple scp -r $folder $host:$folder would suffice), and if that's the case, I wouldn't want to use sshfs. (The alternative would be using rsync over sshfs, defeating the purpose of using sshfs in the first place.) Sep 18, 2013 at 14:25
  • 1
    If the source and destination are both on the same filesystem, the link to the file is modified. If the file is on a different filesystem, the file is first copied and the original file has its link removed. If the connection breaks before the file is copied, you may potentially have a partial file, but the original would be left alone. Sep 18, 2013 at 14:25
  • @Blacklight Shining: depends on what you call unreliable (we use it on a daily basis). The point is that if you let the connection inactive for some time, one endpoint (either client or server) may decide to close the connection. If you perform a sshfs / mv (or cp) / fuserumount, there won't be any trouble, since the connection is always used. If you perform an sshfs in the morning and expect the connection to be up say an hour later, you have to be sure your sshd and possibly the client are configured so that the connection is maintained up. Sep 18, 2013 at 14:36
  • 1
    Hi Morgan. UtahJarhead answered: if the connection breaks for any reason during file transfer, you'll have a partially transfered file on the receiving side, and your file intact on the sending side (since file on the sending side is only deleted after the transfer). Sep 18, 2013 at 16:01
0

You could split the files up into blocks of ten for example, and then

rsync (source files) destination

And, if it ran successfully, follow it with

rsync --remove-source-files (source files) destination

The added bandwidth of a second rsync command is negligible.

2
  • No need to run rsync again; it'd just waste time checking to make sure that the files are all the same—which it just did, in the first invocation. After rsync, you can just do rm -r $source_files. Sep 19, 2013 at 11:17
  • Doh! Good point. I got caught up in the rsync euphoria. It -may- provide a slightly larger sense of security, in that the second rsync checks the work of the first one... but, I think that would be minimal at best.
    – Kent
    Sep 19, 2013 at 23:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .