Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

My question is about saving many files from many directories in only one directories by keeping all old versions of the files. I have about 10 directories with very similar files (10 backup directories made using Linux, Windows and MacOS). I want to concatenate these 10 directories in only one. For example, I have 3 directories a, b and c with the following contents:

  • a

    • fileA.txt (date 10/10/2012)
    • fileB.txt (date 10/12/2012)
    • fileC.txt (date 01/05/2013)
  • b

    • fileA.txt (date 10/01/2013)
    • fileB.txt (date 10/02/2013)
    • fileC.txt (date 01/05/2013)
    • fileD.txt (date 02/05/2013)
  • c

    • fileA.txt (date 10/10/2012)
    • fileB.txt (date 12/02/2013)

The resulting directory must be:

  • fileA.txt (date 10/01/2013 from b)
  • fileA.txt_20121010hhmmss (from a and c (same file))
  • fileB.txt (date 12/02/2013 from c)
  • fileB.txt_20121210hhmmss (from a)
  • fileB.txt_20130210hhmmss (from b)
  • fileC.txt (date 01/05/2013 from a and b (same file))
  • fileD.txt (date 02/05/2013 from b)

hh=hours mm=minutes ss=seconds

Currently I am working on a Perl script which can compare the files using function "compare" and a md5 checksum.

I order to find the modify date (if the files are differents), I am using the Metadata (when I can) using Exiftool (few compatible files like docx, ppt pdf jpg...) otherwise I am using the timestamp using stat command.

The file without the added extension must be the more recent of all the files with the same name.

The Perl script is very slow and complex therefore I am looking for a more robust solution.

Thank you for your help.

NB: - I want to use this solution on a Synology server (DS212) on which I only have really few classical shell commands. If necessary, I can copy these files on a linux (or OSX) machine to do this.

share|improve this question
    
you can keep diffs instead of full files (works best on txt files though) –  ratchet freak Jun 4 '13 at 14:50
    
Have you considered a version-control system such as Git, Subversion, or Mercurial instead? –  chepner Jun 4 '13 at 15:17
    
@ratchetfreak the directories contain pictures, movies, binaries files, Microsoft Office files... Currently the space is not a problem. –  Guuk Jun 4 '13 at 15:21
    
@chepner the problem is that I don't know the more recent files. In version-control system, you add successively and progressively the files (you work on the directories). Each new modification is treated as a new version (I think). Another problem is that the directories contains about one gb of many kind of files (and binaries). –  Guuk Jun 4 '13 at 15:26
    
Maybe one idea is to use rsync command. In fact, when you use the option -u you can avoid to delete the files in the destination but rsync add only a ~ at the end. Moreover, I don't know if rsync is able to taking into account of the timestamp of the file (and more precisely of the modify date) for keeping the more recent file. –  Guuk Jun 4 '13 at 15:30

1 Answer 1

Solution:

cat backup.sh
#!/bin/sh
for i in `find ./{a,b,c} -type f`
do
  FILE=`basename $i`
  FTIME=`stat -c %x $i | awk -F"." '{print $1}' | sed 's/[-: ]//g'`
  NEWNAME=${FILE}_${FTIME}
  echo "cp $i result/${NEWNAME}"
  cp -p $i result/${NEWNAME}
done
cd result
for i in `ls -1 | awk -F"_" '{print $1}' | sort -u`
do
  TNAME=`ls -1t ${i}* | head -1`
  echo "mv ${TNAME} ${i}"
  mv ${TNAME} ${i}
done

Backup files in "result" directory.

How it is working (example):

$ ls -l {a,b,c,result}
a:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileB.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileC.txt

b:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileC.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileD.txt

c:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt

result:
total 0


~/tmp/backup $ ./backup.sh 
cp ./a/fileA.txt result/fileA.txt_20130606162832
cp ./a/fileB.txt result/fileB.txt_20130606162836
cp ./a/fileC.txt result/fileC.txt_20130606162846
cp ./b/fileA.txt result/fileA.txt_20130606162857
cp ./b/fileD.txt result/fileD.txt_20130606162910
cp ./b/fileB.txt result/fileB.txt_20130606162900
cp ./b/fileC.txt result/fileC.txt_20130606162906
cp ./c/fileA.txt result/fileA.txt_20130606162920
cp ./c/fileB.txt result/fileB.txt_20130606162923
mv fileA.txt_20130606162920 fileA.txt
mv fileB.txt_20130606162923 fileB.txt
mv fileC.txt_20130606162906 fileC.txt
mv fileD.txt_20130606162910 fileD.txt


~/tmp/backup $ ls -l {a,b,c,result}
a:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileB.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileC.txt

b:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileC.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileD.txt

c:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt

result:
total 0
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileA.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt_20130606162832
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileA.txt_20130606162857
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileB.txt_20130606162836
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileB.txt_20130606162900
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileC.txt
-rw-r--r-- 1 test test 0 Jun  6 16:28 fileC.txt_20130606162846
-rw-r--r-- 1 test test 0 Jun  6 16:29 fileD.txt


~/tmp/backup $
share|improve this answer
    
Thank you for your answer. In your opinion, which technic can I use to check if the duplicated files exist? –  Guuk Jun 7 '13 at 6:45
    
It depends on your backup system. There can be many nuances. As example you can check MD5 sum for duplicates, or use "diff". –  september Jun 7 '13 at 17:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.