Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have directories with multiple conflicting files, and am looking for a way to sort, name and delete duplicates based on a distinct pattern based on the file's name and size. Though my knowledge of bash scripting is very limited, so I'm looking for any guidance available for how I would create this.

  • The duplicates can be distinct by filenames and filesize
  • If there's a duplicate, the file name ends with _conflict-yyyymmdd-hhmmss.ext
  • There's always an unprefixed version of the file, but this can be corrupt
  • The file with the largest size is always the correct
  • Multiple files can have the same size, in this case the removal of duplicates and renaming of the file is the most desirable
  • The endfile should always be unprefixed

Here's an example of possible scenarios:

   FILE                                                   SIZE        DESIRED ACTION
a. /path/to/dir1/FileName1.ext                            0           rm
b. /path/to/dir1/FileName1_conflict-20130324-231953.ext   21624832    mv b a
c. /path/to/dir1/FileName1_conflict-20130326-080529.ext   21624832    rm

a. /path/to/dir2/FileName2.ext                            25432935    -
b. /path/to/dir2/FileName2_conflict-20130324-092544.ext   0           rm
c. /path/to/dir2/FileName2_conflict-20130326-212307.ext   25432935    rm

a. /path/to/dir3/FileName3.ext                            0           rm
b. /path/to/dir3/FileName3_conflict-20130324-214501.ext   23422234    mv b a

a. /path/to/dir4/FileName4.ext                            0           rm
b. /path/to/dir4/FileName4_conflict-20130324-110541.ext   14423       rm
c. /path/to/dir4/FileName4_conflict-20130326-030512.ext   25432935    mv c a

Any help and suggestions with this would be greatly appreciated.


Update: Solution

Very sorry for not giving any clear examples of what I had tried, my first attempts were a mess. I wasn't looking for a complete script, just some help an guidance. Anyway, I spent the day looking into shell scripting (this is pretty much my first attempt) and I came up with a solution that works for me. You can find my script below:

#!/bin/bash

# find all files without _conflict-suffix
find ./ -type f ! -name "*_conflict*" -print0 | while read -d $'\0' file
do
  # regex to get dir and partial name
  if [[ $file =~ ^(.*\/)(.*)(\..*)$ ]] ; then
    dir="${BASH_REMATCH[1]}"
    # file size in bytes
    size=`wc -c < "$file"`

    # look for matching files with _conflict-suffix
    find "${dir}" -name "${BASH_REMATCH[2]}_conflict-*${BASH_REMATCH[3]}" -print0 | while read -d $'\0' conflict
    do
      # conflicting filesize in bytes
      conSize=`wc -c < "$conflict"`
      # check if conflict-file is bigger
      if [[ $size -lt $conSize ]] ; then
        # It is! Remove, rename, and update variable with new size
        rm "${file}"
        mv "${conflict}" "${file}"
        set size=conSize
      else
        # It isn't, remove the trash.
        rm "${conflict}"
      fi
    done
  fi
done

Take a look at @NSD's answer for another solution.

share|improve this question
2  
what did you try .... did you face any issue's / errors implementing it or Do you want the complete script coded by someone ? (to start with the size can be found using du -sh filename and a if condition to check and rename ... thats what your question requires , right !!) –  nsd Jun 16 '13 at 14:01
    
your example helps... can you rules list be reduced as follows: find set of largest files (by size) in the directory, if unprefixed is in the set, delete all others else move first prefixed large file to unprefixed and delete all others... ??? –  user1055604 Jun 16 '13 at 15:26
    
@SimonKberg +1 for the effort you put in –  nsd Jun 17 '13 at 4:10

1 Answer 1

up vote 0 down vote accepted

hmm .... kinda felt a bit bad that you got down voted , i have something to get you started with.

I read the rules and it seems you already have a description ready on whats to be done for which file ..... the following sample script uses that descriptor to determinee what action is to be taken and then perform that (Note - the mv and rm commands are commented , i was using echo to test , uncomment them to get the actual action)

code ::

Kaizen ~/so_test/test $ cat ztest.sh

## remove header
 sed -i '1d' zlst ;
 :>ztmp ;


## loop to parse the file entries one by one
 while read line
 do

 echo $line ;
 file=`echo $line | tr -s " " | cut -d" " -f2`;
 file=`basename $file .ext` ;   #3 used to filter a selection for action
 echo $file ;

 dir=`echo $line | tr -s " " | cut -d" " -f2`;
 dir=`basename $dir` ;   #3 used to change the directory to where action is to be performed
 cd $dir  &&  echo $dir ;


grep -i $file zlst > ztmp ;
axn=`echo $line | tr -s " " | cut -d" " -f4` ;

 ## case to select the course of axn
 case $axn in

  mv )
      echo `echo $line | tr -s " " | cut -d" " -f4-`;
      file=`echo $line | tr -s " " | cut -d" " -f5` ;
      old_name=`grep -w "${file}" ztmp | cut -d" " -f2` ;
      old_name=`basename $old_name` ;
      file=`echo $line | tr -s " " | cut -d" " -f6` ;
      new_name=`grep -w "${file}" ztmp | cut -d" " -f2` ;
      new_name=`basename $new_name` ;
      echo " mv $old_name $new_name ;" ;
      #mv $old_name $new_name ;
      ## break ;;
      ;;

      rm )
      #file=`echo $line | tr -s " " | cut -d" " -f4-`;
      echo "rm $file" ;
      #rm $file ;
      ## break ;;
      ;;

      * )
       :
       echo "do nothing " ;
       #break ;;
       ;;
  esac

 ##empty the tmp file
 :> ztmp ;

 done < zlst  ## end the while loop @ EOF.

output ::

Kaizen ~/so_test/test $ . ./ztest.sh

a. /path/to/dir1/FileName1.ext 0 rm
FileName1
rm FileName1

b. /path/to/dir1/FileName1_conflict-20130324-231953.ext 21624832 mv b a
FileName1_conflict-20130324-231953
 mv b a
 mv FileName1_conflict-20130324-231953.ext FileName1_conflict-20130324-231953.ext ;

c. /path/to/dir1/FileName1_conflict-20130326-080529.ext 21624832 rm
FileName1_conflict-20130326-080529
rm FileName1_conflict-20130326-080529

.ext
do nothing

a. /path/to/dir2/FileName2.ext 25432935 -
FileName2
do nothing

b. /path/to/dir2/FileName2_conflict-20130324-092544.ext 0 rm
FileName2_conflict-20130324-092544
rm FileName2_conflict-20130324-092544

 c. /path/to/dir2/FileName2_conflict-20130326-212307.ext 25432935 rm
 FileName2_conflict-20130326-212307
 rm FileName2_conflict-20130326-212307

.ext
do nothing

a. /path/to/dir3/FileName3.ext 0 rm
FileName3
rm FileName3

b. /path/to/dir3/FileName3_conflict-20130324-214501.ext 23422234 mv b a
FileName3_conflict-20130324-214501
mv b a
mv FileName3_conflict-20130324-214501.ext FileName3_conflict-20130324-214501.ext ;

.ext
do nothing

a. /path/to/dir4/FileName4.ext 0 rm
FileName4
rm FileName4

b. /path/to/dir4/FileName4_conflict-20130324-110541.ext 14423 rm
FileName4_conflict-20130324-110541
rm FileName4_conflict-20130324-110541

c. /path/to/dir4/FileName4_conflict-20130326-030512.ext 25432935 mv c a
FileName4_conflict-20130326-030512
mv c a
mv FileName4_conflict-20130326-030512.ext FileName4_conflict-20130326-030512.ext ;

Please note : for the size check , i didnt include that in the code , a simple if condition in the while loop will do.... you will need to add that though.

hope this helps !!

share|improve this answer
    
Thanks, this was really helpful. I ended up writing a solution of my own and I'll update the question with it. Will mark yours as the answer though since the question was so unspecific (i did not expect anyone to write a whole script, just give some help an suggestions since I've never written any shell script before). Thanks again –  Simon Kjellberg Jun 16 '13 at 18:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.