1

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., and going up to "(1) (1) (1)".

Do you know of a program or script that can merge these folders?

Example top-level folder structure:

1100 Beetledwarf - Happy ATE
1100 Beetledwarf - Happy ATE (1)
1100 Beetledwarf - Happy ATE (2)
1100 Beetledwarf - Happy ATE (3)
1100 Beetledwarf - Happy ATE (3) (1)
1100 Beetledwarf - Happy ATE (3) (1) (1)
1100 Beetledwarf - Happy ATE (4)
1100 Beetledwarf - Happy ATE (5)
1100 Beetledwarf - Happy ATE (6)

Because subfolders sometimes also have the same problem, the program or script would need to be able to merge folders that follow that naming pattern for all subfolders, example:

Example 2nd level folders:

1100 Beetledwarf - Happy ATE (6)
    Analysis
    Analysis (1)
    Smirckle_HL
    Smirckle_HL (2)
    Pending Reports
    Photos & Logos

The best solution would also allow me to move files instead of copying them since it takes a long time to copy files but moving is almost instantaneous.

List of things I've already tried, but none of them can deal with the "name (1)" folder structure (that I can tell so far), and all of them copy files instead of moving them:

  • WinMerge for Windows 10 <- chokes when trying to copy google drive files (returns something like "DOS command not supported" for them)
  • Meld for MacOS. <- slow.
  • Terminal with "ditto" command in OS X <- Best option so far.

Thanks for your help!

  • If I understood you correctly, moving files in your case through the internet connection will take the same time as copying them. You confuse it with moving files for example within a HD-partition, that's quick. Moving them between partitions, between HDs, between PCs etc. will usually take considerably more time (not considering exception with certain configurations). – Albin Jul 11 '18 at 17:48
  • Thanks, I removed the bit about my internet connection because it's extraneous. When using google file stream in Windows and Mac OS, if you move files, it's like moving files within an HD-partition: if you move files, the OS usually just changes the pointers that point to the file, but if you copy them, then the OS usually copies the data to a new location on the drive. In this case, the copy takes even more time because the HD is attached over the internet. Cheers! – Josh Jul 12 '18 at 21:58
  • Hi there. I'm working on a pyhton script to solve specifically your question. It's reasonably ready, but unfortunately I can't spend more time making the (important) finishing touches (safeguards) until monday, =/ since I will be travelling. If you stumble upon an answer before that, well - great for you anyway =D, but if you don't find any, I can finish this and post it here as an answer =). Have a good day. – Vinícius M Jul 13 '18 at 2:18
1

This is the approach I would try in Linux. I have no experience with Google Filestream, Google Drive nor Synology CloudSync, so I cannot tell if the solution can be applied at all. Still I hope this will at least give you some ideas.


Assumptions

  • you can mount the share in your directory tree, so mv, cp and other sane tools can work with directories as if they were local;
  • files (or directories) with paths that become identical after you remove all (N) strings are in fact instances of the same file (directory);
  • instances of the same file should leave just one file;
  • instances of the same directory should merge their content in a single directory;
  • you can use all the tools I use here.

Procedure

Please read the entire answer before attempting to do anything.

I think some steps could be written as a script, but since the solution is highly experimental, it's better to do it by hand, step by step, paying attention what happens.

  1. In a shell cd to the mountpoint and invoke find . | vidir -; use a text editor of your choice, e.g. kate, like this:

    find . | EDITOR=kate vidir
    

    This will open the editor with a list of all objects, each one with its own number in front. When you alter the content and save the (temporary) file and close the editor, all the changes are applied. In general this is what you can do:

    • change paths to move (rename) files or directories;
    • delete lines to remove files or directories;
    • swap two or more numbers to swap files (you won't need it).

    Don't save the file unless you're sure the new content describes the directory tree you want to get.

  2. Copy the content from the editor to another file. The point is to work with it and paste the result back (and save it) only when you're sure you got it right. Next steps refer to the new file unless explicitly stated otherwise.

  3. Use sed or any other tool to get rid of all (N) strings (note the leading space). At this point you should get "clean" paths, many of them will occur more than once (with different numbers given by vidir).

  4. Use sort -k 2 to sort according to these paths. Thanks to -s the former Analysis should still precede the former Analysis (1).

  5. Use uniq -f 1 to drop duplicated paths. Now any path should occur just once.

  6. Double check the sanity of the directory structure encoded in the result.

  7. Paste the result into the original editor, save the file and exit the editor. vidir will remove objects associated with missing numbers and move objects associated with numbers that are left.


Testing

I would first use this solution to replicate the directory structure:

cp -a --attributes-only /mountpoint/ /guinea_pig_dir/

and test the procedure on the resulting empty files. This should reveal problems (if any) and hopefully allow to improve the method.


Possible problems

  1. vidir refuses to work with some non-standard characters.

  2. In general the order of objects is important. There are few pitfalls which generate objects like foo~ or foo~1, foo~2 when there's a collision with foo. You will "contract" your directory tree in a way that should generate no collisions, still I haven't investigated all possible scenarios. I really think you should experiment with /guinea_pig_dir/ and see what you get. In case of troubles maybe a clever sort between find and vidir will help.

1

Below is a bash script that performs this task. It works on e.g. MSYS2 Bash with rsync added. It is taken from this related question here:

Script for deduplicating files and folders with a particular suffix

#!/usr/bin/bash
IFS=$'\n';
set -f
#Go deepest first to deal with copies within copied folders.
for copy in $(find . -regextype posix-egrep -regex "^.*\ \([0-9]+\)\s*(\.[^/.]*)?$" | awk '{print length($0)"\t"$0}' | sort -rnk1 | cut -f2-); do
    orig=$(rev <<< "$copy" | sed -E 's/\)[0-9]+\(\ //' | rev)
    if [ "$orig" != "$copy" ]; then
        if [ -f "$orig" ]; then
            if [ -f "$copy" ]; then
                echo "File pair: $orig $copy"
                if diff -q "$orig" "$copy" &>/dev/null; then
                    echo "Removing file: $copy"
                    rm -f "$copy";
                fi
            fi           
        fi
        if [ -d "$orig" ]; then
            if [ -d "$copy" ]; then
                echo "Folder pair: $orig $copy"
                if rmdir "$copy" &>/dev/null; then
                    #If the "copy" was an empty directory then we've removed it and so we're done.
                    echo "Removed empty folder: $copy"
                else
                    #Non-destructively ensure that both folders have the same files at least.                    
                    rsync -aHAv --ignore-existing "$orig/" "$copy" &>/dev/null
                    rsync -aHAv --ignore-existing "$copy/" "$orig" &>/dev/null
                    if diff -qr "$orig" "$copy" &>/dev/null; then
                        echo "Removing folder: $copy"
                        rm -rf "$copy";
                    fi            
                fi
            fi
        fi
    fi
done
unset IFS;
set +f
  • How would I set a starting directory in this script? Both when I test it and in the final roll-out I'll only want it to work with a specific subset of my files. Example: 'G:\My Drive\Deduplicate_Test_Folder' PS: Thank you for the incredible response! – Josh Nov 21 '18 at 22:47
  • Just start in that folder. (I.e. cd there.) – cfp Nov 22 '18 at 20:27
  • I tried this in MSYS2, and got the following error: "find: failed to read file names from file system at or below ‘.’: No such file or directory' – Josh Apr 5 at 18:04
  • update - the error seems to be related to google filestream, because it doesn't happen when I use a folder on a local disk – Josh Apr 5 at 23:04
0

The following script works in OS X, with some issues:

Sometimes Google docs don't copy, and therefore the folders fail the diff test, and they aren't deleted. I then have to manually run diff on them, check that the differences are all Google docs, manually move the google docs into the original folder if they're not there, and then manually delete the duplicate folders. (Note: from finder I can't copy those same documents, which seems weird to me because I feel like I've copied (or option+dragged) google docs files before.)

#!/usr/bin/bash
IFS=$'\n';
set -f
#Go deepest first to deal with copies within copied folders.
for copy in $(find -E . -regex "^.*\ \([0-9]+\)\s*(\.[^/.]*)?$" | awk '{print length($0)"\t"$0}' | sort -rnk1 | cut -f2-); do
    orig=$(rev <<< "$copy" | sed -E 's/\)[0-9]+\(\ //' | rev)
    if [ "$orig" != "$copy" ]; then
        if [ -f "$orig" ]; then
            if [ -f "$copy" ]; then
                echo "File pair: $orig $copy"
                if diff -q "$orig" "$copy" &>/dev/null; then
                    echo "Removing file: $copy"
                    rm -f "$copy";
                fi
            fi           
        fi
        if [ -d "$orig" ]; then
            if [ -d "$copy" ]; then
                echo "Folder pair: $orig $copy"
                if rmdir "$copy" &>/dev/null; then
                    #If the "copy" was an empty directory then we've removed it and so we're done.
                    echo "Removed empty folder: $copy"
                else
                    #Non-destructively ensure that both folders have the same files at least.                    
                    rsync -aHAv --ignore-existing "$orig/" "$copy" &>/dev/null
                    rsync -aHAv --ignore-existing "$copy/" "$orig" &>/dev/null
                    if diff -x ‘.*’ -x 'Icon?' -qr "$orig" "$copy" &>/dev/null; then
                        echo "Removing folder: $copy"
                        trash -v "$copy"      # requires that Ali Rantakari's app is installed: aka that you have already run 'brew install trash'
                        #replaced the following: rm -rf "$copy";
                    fi            
                fi
            fi
        fi
    fi
done
unset IFS;
set +f

Note: this working may rely on the following tools being installed, as detailed on this page: https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x/

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install coreutils
export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH"
brew tap homebrew/dupes
brew install binutils
brew install diffutils
brew install ed --with-default-names
brew install findutils --with-default-names
brew install gawk
brew install gnu-indent --with-default-names
brew install gnu-sed --with-default-names
brew install gnu-tar --with-default-names
brew install gnu-which --with-default-names
brew install gnutls
brew install grep --with-default-names
brew install gzip
brew install screen
brew install watch
brew install wdiff --with-gettext
brew install wget
brew install bash
brew install rsync

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.