Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have directories with multiple conflicting files, and am looking for a way to sort, name and delete duplicates based on a distinct pattern based on the file's name and size. Though my knowledge of bash scripting is very limited, so I'm looking for any guidance available for how I would create this.

  • The duplicates can be distinct by filenames and filesize
  • If there's a duplicate, the file name ends with _conflict-yyyymmdd-hhmmss.ext
  • There's always an unprefixed version of the file, but this can be corrupt
  • The file with the largest size is always the correct
  • Multiple files can have the same size, in this case the removal of duplicates and renaming of the file is the most desirable
  • The endfile should always be unprefixed

Here's an example of possible scenarios:

   FILE                                                   SIZE        DESIRED ACTION
a. /path/to/dir1/FileName1.ext                            0           rm
b. /path/to/dir1/FileName1_conflict-20130324-231953.ext   21624832    mv b a
c. /path/to/dir1/FileName1_conflict-20130326-080529.ext   21624832    rm

a. /path/to/dir2/FileName2.ext                            25432935    -
b. /path/to/dir2/FileName2_conflict-20130324-092544.ext   0           rm
c. /path/to/dir2/FileName2_conflict-20130326-212307.ext   25432935    rm

a. /path/to/dir3/FileName3.ext                            0           rm
b. /path/to/dir3/FileName3_conflict-20130324-214501.ext   23422234    mv b a

a. /path/to/dir4/FileName4.ext                            0           rm
b. /path/to/dir4/FileName4_conflict-20130324-110541.ext   14423       rm
c. /path/to/dir4/FileName4_conflict-20130326-030512.ext   25432935    mv c a

Any help and suggestions with this would be greatly appreciated.

Update: Solution

Very sorry for not giving any clear examples of what I had tried, my first attempts were a mess. I wasn't looking for a complete script, just some help an guidance. Anyway, I spent the day looking into shell scripting (this is pretty much my first attempt) and I came up with a solution that works for me. You can find my script below:


# find all files without _conflict-suffix
find ./ -type f ! -name "*_conflict*" -print0 | while read -d $'\0' file
  # regex to get dir and partial name
  if [[ $file =~ ^(.*\/)(.*)(\..*)$ ]] ; then
    # file size in bytes
    size=`wc -c < "$file"`

    # look for matching files with _conflict-suffix
    find "${dir}" -name "${BASH_REMATCH[2]}_conflict-*${BASH_REMATCH[3]}" -print0 | while read -d $'\0' conflict
      # conflicting filesize in bytes
      conSize=`wc -c < "$conflict"`
      # check if conflict-file is bigger
      if [[ $size -lt $conSize ]] ; then
        # It is! Remove, rename, and update variable with new size
        rm "${file}"
        mv "${conflict}" "${file}"
        set size=conSize
        # It isn't, remove the trash.
        rm "${conflict}"

Take a look at @NSD's answer for another solution.

share|improve this question
what did you try .... did you face any issue's / errors implementing it or Do you want the complete script coded by someone ? (to start with the size can be found using du -sh filename and a if condition to check and rename ... thats what your question requires , right !!) – n13 Jun 16 '13 at 14:01
your example helps... can you rules list be reduced as follows: find set of largest files (by size) in the directory, if unprefixed is in the set, delete all others else move first prefixed large file to unprefixed and delete all others... ??? – user1055604 Jun 16 '13 at 15:26
@SimonKberg +1 for the effort you put in – n13 Jun 17 '13 at 4:10
up vote 0 down vote accepted

hmm .... kinda felt a bit bad that you got down voted , i have something to get you started with.

I read the rules and it seems you already have a description ready on whats to be done for which file ..... the following sample script uses that descriptor to determinee what action is to be taken and then perform that (Note - the mv and rm commands are commented , i was using echo to test , uncomment them to get the actual action)

code ::

Kaizen ~/so_test/test $ cat

## remove header
 sed -i '1d' zlst ;
 :>ztmp ;

## loop to parse the file entries one by one
 while read line

 echo $line ;
 file=`echo $line | tr -s " " | cut -d" " -f2`;
 file=`basename $file .ext` ;   #3 used to filter a selection for action
 echo $file ;

 dir=`echo $line | tr -s " " | cut -d" " -f2`;
 dir=`basename $dir` ;   #3 used to change the directory to where action is to be performed
 cd $dir  &&  echo $dir ;

grep -i $file zlst > ztmp ;
axn=`echo $line | tr -s " " | cut -d" " -f4` ;

 ## case to select the course of axn
 case $axn in

  mv )
      echo `echo $line | tr -s " " | cut -d" " -f4-`;
      file=`echo $line | tr -s " " | cut -d" " -f5` ;
      old_name=`grep -w "${file}" ztmp | cut -d" " -f2` ;
      old_name=`basename $old_name` ;
      file=`echo $line | tr -s " " | cut -d" " -f6` ;
      new_name=`grep -w "${file}" ztmp | cut -d" " -f2` ;
      new_name=`basename $new_name` ;
      echo " mv $old_name $new_name ;" ;
      #mv $old_name $new_name ;
      ## break ;;

      rm )
      #file=`echo $line | tr -s " " | cut -d" " -f4-`;
      echo "rm $file" ;
      #rm $file ;
      ## break ;;

      * )
       echo "do nothing " ;
       #break ;;

 ##empty the tmp file
 :> ztmp ;

 done < zlst  ## end the while loop @ EOF.

output ::

Kaizen ~/so_test/test $ . ./

a. /path/to/dir1/FileName1.ext 0 rm
rm FileName1

b. /path/to/dir1/FileName1_conflict-20130324-231953.ext 21624832 mv b a
 mv b a
 mv FileName1_conflict-20130324-231953.ext FileName1_conflict-20130324-231953.ext ;

c. /path/to/dir1/FileName1_conflict-20130326-080529.ext 21624832 rm
rm FileName1_conflict-20130326-080529

do nothing

a. /path/to/dir2/FileName2.ext 25432935 -
do nothing

b. /path/to/dir2/FileName2_conflict-20130324-092544.ext 0 rm
rm FileName2_conflict-20130324-092544

 c. /path/to/dir2/FileName2_conflict-20130326-212307.ext 25432935 rm
 rm FileName2_conflict-20130326-212307

do nothing

a. /path/to/dir3/FileName3.ext 0 rm
rm FileName3

b. /path/to/dir3/FileName3_conflict-20130324-214501.ext 23422234 mv b a
mv b a
mv FileName3_conflict-20130324-214501.ext FileName3_conflict-20130324-214501.ext ;

do nothing

a. /path/to/dir4/FileName4.ext 0 rm
rm FileName4

b. /path/to/dir4/FileName4_conflict-20130324-110541.ext 14423 rm
rm FileName4_conflict-20130324-110541

c. /path/to/dir4/FileName4_conflict-20130326-030512.ext 25432935 mv c a
mv c a
mv FileName4_conflict-20130326-030512.ext FileName4_conflict-20130326-030512.ext ;

Please note : for the size check , i didnt include that in the code , a simple if condition in the while loop will do.... you will need to add that though.

hope this helps !!

share|improve this answer
Thanks, this was really helpful. I ended up writing a solution of my own and I'll update the question with it. Will mark yours as the answer though since the question was so unspecific (i did not expect anyone to write a whole script, just give some help an suggestions since I've never written any shell script before). Thanks again – Simon Kjellberg Jun 16 '13 at 18:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .