Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have this output from a s3cmd ls --list-md5 command.

    2013-05-22 12:20   1965154   41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz
    2013-05-22 12:20  11456836   541b1bf78682f48867cc99dbb53c4c3a  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121118040001.gz

I'm trying to verify if the md5sum from my object in the bucket match with my md5sum in my local machine. So for that I need to get the hash result as the fourth column and the filename as the fifth column.

Like that:

41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz

But I need only the filename, so for that I'm using the basename command (I just don't know how to use it here yet but I will figure out), at the end is looking like this:

s3=`s3cmd ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/`;
echo "$s3";

echo "$s3" |  tr -s ' ' | cut -d ' ' -f 4;
echo "$s3" |  tr -s ' ' | cut -d ' ' -f 5;

Now I need to store the result from the echo's at the same variable like these:

41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz

So I could compare the results from s3cmd ls --list-md5sum with my md5sum *.gz running locally.

update Using the diff approach my command looks like:

ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup/server245/ \
| tr -s ' ' \
| cut -d ' ' -f 4,5 \
| sed 's= .*/= =' \
| diff -w - <(md5sum /backup/server245/*.gz)

Giving the follow result:

< d41d8cd98f00b204e9800998ecf8427e 
< 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
< 541b1bf78682f48867cc99dbb53c4c3a app.20121118040001.gz
> 41eae9b40d23de2f02bf07635870f6d0  /backup/diveo245/app.20121117040001.gz
> 541b1bf78682f48867cc99dbb53c4c3a  /backup/diveo245/app.20121118040001.gz

But how to know the file that is different then ? ('cause all that are equal I can delete alredy)

share|improve this question
can't you just Var=$(echo....)? – demure May 22 '13 at 13:51
up vote 5 down vote accepted

You need no variables. You can use diff to compare the outputs of two different commands:

s3cmd ls --list-md5 \
    | tr -s ' ' \
    | cut -d ' ' -f 5,6 \
    | sed 's= .*/= =' \
    | diff -w - <(md5sum *.gz)

You might need to sort the outputs to have the files in the same order.

share|improve this answer
Or omit the diff part, write the output to a file like s3.md5 and use md5sum -c s3.md5 to check against your local files. – mpy May 22 '13 at 14:06
I just don't understand the output of the diff and how to handle with it. I update my post with the output. – Valter Henrique May 22 '13 at 15:03
@ValterHenrique: In your original question, you were running md5sum *.gz. If you include path to the md5sum arguments, you have to delete it from its output, or cd /backup/server245/*.gz and use the command I provided. – choroba May 22 '13 at 15:05
got it, thank you for you help friend! It really help me out. – Valter Henrique May 22 '13 at 15:10

@choroba's answer is the way to go, but here is another way using variables as requested:

s3=`s3cmd ls --list-md5 \
 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/ | 
 awk '{print $4,$5}' | awk -F"[/ ]" '{print $1,$NF}'`
echo $s3
 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
share|improve this answer
+1 for the awk -F ... $NF trick. – mpy May 22 '13 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .