I have two directories with thousands of files which contain more or less the same files.

How can I copy all files from dirA to dirB which are not in dirB or if the file exists in dirB only overwrite it if it's smaller.

I know there are a lot of examples for different timestamp or different file size but I only want to overwrite if the destination file is smaller and under no circumstances if it's bigger.

Background of my problem:
I've rendered a dynmap on my Minecraft Server but some of the tiles are missing or corrupted. Then I did the rendering again on another machine with a faster CPU and copied all the new rendered files (~50GB / 6.000.000 ~4-10 KB PNGs) on my server. After that I noticed that there are also corrupted files in my new render.

left: old render, right: new render

old 1 corrupted new 1

old 2 new 2 corrupted

Therefor I don't want to overwrite all files but only the ones which are bigger (the corrupted carry less data and are smaller).

  • Use cp with combination of cmp commands or better use rsync that has all options you want
    – Alex
    Feb 3, 2017 at 12:34
  • What option do I have to use with rsync? I didn't find anything for larger files only newer or different size. That's why I asked.
    – das Keks
    Feb 3, 2017 at 12:47
  • Use stat on files in both locations to get files size and then copy if it satisfy your conditions then
    – Alex
    Feb 3, 2017 at 13:02
  • Well, it a challenge, looked for rsync options you need but fail to find right one, so went with a simple way
    – Alex
    Feb 3, 2017 at 23:28

5 Answers 5


May be a dirty way, but I hope it is what you are looking for


### Purpose:
# Copy huge amount of files from source to destination directory only if
# destination file is smaller in size than in source directory

src='./d1' # Source directory
dst='./d2' # Destination directory

icp() {
  [ -d "$f" ] && {
    [ ! -d "${dst}${f#$src}" ] && mkdir -p "${dst}${f#$src}";

  [ ! -f "${dst}/${f#$src/}" ] && { cp -a "${f}" "${dst}/${f#$src/}"; return; }
  fsizeSrc=$( stat -c %s "$f" )
  fsizeDst=$( stat -c %s "${dst}/${f#$src/}" )
  [ ${fsizeDst} -lt ${fsizeSrc} ] && cp -a "${f}" "${dst}/${f#$src/}"

export -f icp
export src
export dst

find ${src} -exec bash -c 'icp "$0"' {} \;
  • Thanks. I tested it with some test data and it works as I need it. But when I want to execute it on my real data I have a problem because the directory contains too many files (about 6.000.000) :ls argument list too long)
    – das Keks
    Feb 4, 2017 at 9:29
  • This is operation system limit (you can get it for your system as : getconf ARG_MAX). You probably have there pretty long file names or very deep directories structure, so when find feed ls with such names it exceed maximum allowed length for command line. I modified a little script to eliminate ls command, could you try this new version.
    – Alex
    Feb 4, 2017 at 9:46
  • If script would choke again, you may try to reduce full path by mounting it to some short path. For example sudo mkdir -m 777 /a then mount source directory to /a as sudo mount --bind /pretty/long/prefix/to/source/directory /a then use /a in my script. When you done, unmount /a by issue command: sudo umount /a
    – Alex
    Feb 4, 2017 at 10:09
  • I think it's not the path length since the longest path (including file name) is about 80 characters long. Could it be the list, which is passed to the for each, which is too long? I think this question targets something similar: unix.stackexchange.com/questions/128559/…
    – das Keks
    Feb 4, 2017 at 10:59
  • May be diff --brief -r dir1/ dir2/ is a good approach and then do something for each line of the output. I'll try to construct something like this in the evening.
    – das Keks
    Feb 4, 2017 at 11:20

My problem had been similar. I wanted to synchronize files from a remote folder to a local one, but only copy the remote files which were bigger than the according local files.

My workaround with rsync was like that, which in fact a bash one-liner:

for x in $(ls -1 home/me/local/folder/*)
    eachsize=$(stat -c "%s")
    rsync -avz --progress --max-size=${eachsize} remote:/home/you/folder/${x} .

I think you can get the point, since the filenames are the same between the two folders, I go through each one in the local folder and keep its size, then I place it as a limit whether rsync should copy or not the remote file of the same name but different size.

  • Don’t use ls like that; just do for x in home/me/local/folder/*. Sep 4, 2017 at 13:01
  • You are right; just to make my point though.
    – user32916
    Sep 4, 2017 at 15:42
  • testing this approach I think using ls like that or not depends on how you want empty folders handled -- in such cases for x in $(ls folder/*) will execute the loop zero times, whereas for x in folder/* will execute the loop precisely one time (against the non-expandable folder/* literal as-if a filename)
    – humbletim
    Nov 24, 2019 at 17:03

You can use rsync command

Syntax :

-a = archive mode
-v = increase verbosity
-z = compress file data during the transfer
--progress = show progress during transfer

rsync -avz --progress <source path> <destination path>

you can use --delete to delete extraneous files from destination directory

rsync -avz --delete --progress <source path> <destination path>

so your command will be:

rsync -avz --delete --progress dirA dirB
  • 1
    Doesn't the -a flag copies all files which have newer timestamp or different file size? It's important that only smaller files will be overwritten.
    – das Keks
    Feb 3, 2017 at 12:57
  • this command will not overwrite anything, this will copy only changed file and new file which is not available under Destination Director. Feb 3, 2017 at 21:17
  • Changed files will be overwritten in the destination. Regardless of the size of the destination file. Tested it with some data and the -a option is not what I need.
    – das Keks
    Feb 3, 2017 at 21:31

rsync -va --append dirA dirB

just got from man rsync, and you can get more of what you want.

  • --append: append data onto shorter files
  • -v verbose
  • -a archive mode; same as -rlptgoD (no -H)

--append is very useful for me when the cp -ru a b process was interrupted several times and you also changed the updated time of files by chown user:user -R *. haha :)


I've modified this to something like:

# Copy src to destination if the src is larger.
function copy_if_larger() {
  local src="$1"
  local dest="$2"

  [ ! -f "$1" ] return
  [ ! -f "$2" ] return

  local srcSize=$( stat -c %s "$1")
  local dstSize=$( stat -c %s "$2")

  [ ${dstSize} -lt ${srcSize} ] && {
    cp -a "$1" "$2"

Then I wrote another method to adjust the files that I want to copy and feed them into the copy_if_larger function.

function do_copy_if_larger() {
  # trim prefix
  local suffix=$(echo "$1" | cut -c 10-)
  copy_if_larger "$1" "/dest/path/$suffix"

# make the functions visible to the subshell.
export -f copy_if_larger
export -f do_copy_if_larger

# copy all larger jpeg files over /dest/path
find . -name '*jpg' | xargs -n 1 bash -c 'do_copy_if_larger "$@"' {}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.