1

I have this output from a s3cmd ls --list-md5 command.

    2013-05-22 12:20   1965154   41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz
    2013-05-22 12:20  11456836   541b1bf78682f48867cc99dbb53c4c3a  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121118040001.gz
    ..

I'm trying to verify if the md5sum from my object in the bucket match with my md5sum in my local machine. So for that I need to get the hash result as the fourth column and the filename as the fifth column.

Like that:

41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz

But I need only the filename, so for that I'm using the basename command (I just don't know how to use it here yet but I will figure out), at the end is looking like this:

s3=`s3cmd ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/`;
echo "$s3";

echo "$s3" |  tr -s ' ' | cut -d ' ' -f 4;
echo "$s3" |  tr -s ' ' | cut -d ' ' -f 5;

Now I need to store the result from the echo's at the same variable like these:

41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz

So I could compare the results from s3cmd ls --list-md5sum with my md5sum *.gz running locally.

update Using the diff approach my command looks like:

ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup/server245/ \
| tr -s ' ' \
| cut -d ' ' -f 4,5 \
| sed 's= .*/= =' \
| diff -w - <(md5sum /backup/server245/*.gz)

Giving the follow result:

1,50c1,49
< d41d8cd98f00b204e9800998ecf8427e 
< 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
< 541b1bf78682f48867cc99dbb53c4c3a app.20121118040001.gz
..
---
> 41eae9b40d23de2f02bf07635870f6d0  /backup/diveo245/app.20121117040001.gz
> 541b1bf78682f48867cc99dbb53c4c3a  /backup/diveo245/app.20121118040001.gz
..

But how to know the file that is different then ? ('cause all that are equal I can delete alredy)

1
  • 1
    can't you just Var=$(echo....)?
    – demure
    May 22, 2013 at 13:51

2 Answers 2

5

You need no variables. You can use diff to compare the outputs of two different commands:

s3cmd ls --list-md5 \
    | tr -s ' ' \
    | cut -d ' ' -f 5,6 \
    | sed 's= .*/= =' \
    | diff -w - <(md5sum *.gz)

You might need to sort the outputs to have the files in the same order.

4
  • 2
    Or omit the diff part, write the output to a file like s3.md5 and use md5sum -c s3.md5 to check against your local files.
    – mpy
    May 22, 2013 at 14:06
  • I just don't understand the output of the diff and how to handle with it. I update my post with the output. May 22, 2013 at 15:03
  • 1
    @ValterHenrique: In your original question, you were running md5sum *.gz. If you include path to the md5sum arguments, you have to delete it from its output, or cd /backup/server245/*.gz and use the command I provided.
    – choroba
    May 22, 2013 at 15:05
  • got it, thank you for you help friend! It really help me out. May 22, 2013 at 15:10
1

@choroba's answer is the way to go, but here is another way using variables as requested:

s3=`s3cmd ls --list-md5 \
 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/ | 
 awk '{print $4,$5}' | awk -F"[/ ]" '{print $1,$NF}'`
echo $s3
 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
1
  • +1 for the awk -F ... $NF trick.
    – mpy
    May 22, 2013 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .