2

I have two directories with thousands of files which contain more or less the same files.

How can I copy all files from dirA to dirB which are not in dirB or if the file exists in dirB only overwrite it if it's smaller.

I know there are a lot of examples for different timestamp or different file size but I only want to overwrite if the destination file is smaller and under no circumstances if it's bigger.

Background of my problem:
I've rendered a dynmap on my Minecraft Server but some of the tiles are missing or corrupted. Then I did the rendering again on another machine with a faster CPU and copied all the new rendered files (~50GB / 6.000.000 ~4-10 KB PNGs) on my server. After that I noticed that there are also corrupted files in my new render.

left: old render, right: new render

old 1 corrupted new 1

old 2 new 2 corrupted

Therefor I don't want to overwrite all files but only the ones which are bigger (the corrupted carry less data and are smaller).

  • Use cp with combination of cmp commands or better use rsync that has all options you want – Alex Feb 3 '17 at 12:34
  • What option do I have to use with rsync? I didn't find anything for larger files only newer or different size. That's why I asked. – das Keks Feb 3 '17 at 12:47
  • Use stat on files in both locations to get files size and then copy if it satisfy your conditions then – Alex Feb 3 '17 at 13:02
  • Well, it a challenge, looked for rsync options you need but fail to find right one, so went with a simple way – Alex Feb 3 '17 at 23:28
2

May be a dirty way, but I hope it is what you are looking for

#!/bin/bash

### Purpose:
# Copy huge amount of files from source to destination directory only if
# destination file is smaller in size than in source directory
###

src='./d1' # Source directory
dst='./d2' # Destination directory

icp() {
  f="${1}";
  [ -d "$f" ] && {
    [ ! -d "${dst}${f#$src}" ] && mkdir -p "${dst}${f#$src}";
    return
  }

  [ ! -f "${dst}/${f#$src/}" ] && { cp -a "${f}" "${dst}/${f#$src/}"; return; }
  fsizeSrc=$( stat -c %s "$f" )
  fsizeDst=$( stat -c %s "${dst}/${f#$src/}" )
  [ ${fsizeDst} -lt ${fsizeSrc} ] && cp -a "${f}" "${dst}/${f#$src/}"
}

export -f icp
export src
export dst

find ${src} -exec bash -c 'icp "$0"' {} \;
  • Thanks. I tested it with some test data and it works as I need it. But when I want to execute it on my real data I have a problem because the directory contains too many files (about 6.000.000) :ls argument list too long) – das Keks Feb 4 '17 at 9:29
  • This is operation system limit (you can get it for your system as : getconf ARG_MAX). You probably have there pretty long file names or very deep directories structure, so when find feed ls with such names it exceed maximum allowed length for command line. I modified a little script to eliminate ls command, could you try this new version. – Alex Feb 4 '17 at 9:46
  • If script would choke again, you may try to reduce full path by mounting it to some short path. For example sudo mkdir -m 777 /a then mount source directory to /a as sudo mount --bind /pretty/long/prefix/to/source/directory /a then use /a in my script. When you done, unmount /a by issue command: sudo umount /a – Alex Feb 4 '17 at 10:09
  • I think it's not the path length since the longest path (including file name) is about 80 characters long. Could it be the list, which is passed to the for each, which is too long? I think this question targets something similar: unix.stackexchange.com/questions/128559/… – das Keks Feb 4 '17 at 10:59
  • May be diff --brief -r dir1/ dir2/ is a good approach and then do something for each line of the output. I'll try to construct something like this in the evening. – das Keks Feb 4 '17 at 11:20
1

You can use rsync command

Syntax :

-a = archive mode
-v = increase verbosity
-z = compress file data during the transfer
--progress = show progress during transfer

rsync -avz --progress <source path> <destination path>

you can use --delete to delete extraneous files from destination directory

rsync -avz --delete --progress <source path> <destination path>

so your command will be:

rsync -avz --delete --progress dirA dirB
  • 1
    Doesn't the -a flag copies all files which have newer timestamp or different file size? It's important that only smaller files will be overwritten. – das Keks Feb 3 '17 at 12:57
  • this command will not overwrite anything, this will copy only changed file and new file which is not available under Destination Director. – Pankaj Jackson Feb 3 '17 at 21:17
  • Changed files will be overwritten in the destination. Regardless of the size of the destination file. Tested it with some data and the -a option is not what I need. – das Keks Feb 3 '17 at 21:31
0

My problem had been similar. I wanted to synchronize files from a remote folder to a local one, but only copy the remote files which were bigger than the according local files.

My workaround with rsync was like that, which in fact a bash one-liner:

for x in $(ls -1 home/me/local/folder/*)
do
    eachsize=$(stat -c "%s")
    rsync -avz --progress --max-size=${eachsize} remote:/home/you/folder/${x} .
done

I think you can get the point, since the filenames are the same between the two folders, I go through each one in the local folder and keep its size, then I place it as a limit whether rsync should copy or not the remote file of the same name but different size.

  • Don’t use ls like that; just do for x in home/me/local/folder/*. – G-Man Sep 4 '17 at 13:01
  • You are right; just to make my point though. – user32916 Sep 4 '17 at 15:42

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.