2

I have two machines, nearbyMachine and remoteMachine.

On nearbyMachine, files are being created in a folder.

nearbyMachine/
    myFiles/
        file1
        file2
        file3
        ...

I want to have the files on remoteMachine as well -- but sorted into directories:

remoteMachine/
    filesByCategory/
        loveLetters/
            file1
            file2
            ...
        otherLetters/
            file3
            ...
        ...

So I...

  • Copy all the files from the folder to a temporary folder on remoteMachine over the Internet,
  • Run a software on remoteMachine that removes all files from the temporary folder, that are already present in any subfolder of filesByCategory.
  • Manually sort the files into existing or new subfolders of filesByCategory.

The problem: As time goes by, the first step takes more and more time. Hours. And it is inelegant.

Is there any way, for example with rsync, that I can copy only the files that are not already present anywhere on the remote machine? Or at least get a list of the files I have to copy?

In my understanding, all synchronization programs can only compare one single directory on both sides.

8
  • A bruteforce way is to use hard or softlinks so you can have a flattened folder on the target machine without taking up much space
    – Peter
    Apr 27, 2019 at 19:27
  • Developing @Peter's idea, why not let rsync do its normal mirroring, then use links to create the additional, independent directory structure on the remote machine?
    – AFH
    Apr 27, 2019 at 19:35
  • Hm, that would have been an interesting idea, if I had had it before I already had hundreds of files, sorted away in countless hours. :-(
    – Kurtibert
    Apr 27, 2019 at 19:37
  • Maybe syncing isn’t what you want. Why not copy files based on their modified / created date? New files will be copied. Old files will not. Apr 27, 2019 at 19:47
  • A faster procedure would be to selectively copy to the temporary folder on remoteMachine only files that are newer than the last sync. Then you will only need to move a few files to their new folders. (Add to your comment @harrymc for me to be notified.)
    – harrymc
    Apr 27, 2019 at 20:50

1 Answer 1

0

A faster procedure would be to selectively copy to the temporary folder on remoteMachine only files that are newer than the last sync. Then you will only need to sort a few files to their folders.

A simple script would do the job. You would use a utility such as find to list all the newer files. The newerXY parameter would perhaps be useful, together with a reference file that your script will touch at the end of the copy.

-newerXY reference
   Succeeds if timestamp X of the file being considered is newer
   than timestamp Y of the file reference.  The letters X and Y
   can be any of the following letters:

     a   The access time of the file reference
     B   The birth time of the file reference
     c   The inode status change time of reference
     m   The modification time of the file reference
     t   reference is interpreted directly as a time

The parameter to use may be -newermm.

Your script could use this list of files generated by find as the parameter to a cp command, to copy to the remote server, where you may sort them to their respective folders.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .