Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have this output from a s3cmd ls --list-md5 command.

    2013-05-22 12:20   1965154   41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz
    2013-05-22 12:20  11456836   541b1bf78682f48867cc99dbb53c4c3a  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121118040001.gz

I'm trying to verify if the md5sum from my object in the bucket match with my md5sum in my local machine. So for that I need to get the hash result as the fourth column and the filename as the fifth column.

Like that:

41eae9b40d23de2f02bf07635870f6d0  s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/app.20121117040001.gz

But I need only the filename, so for that I'm using the basename command (I just don't know how to use it here yet but I will figure out), at the end is looking like this:

s3=`s3cmd ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/`;
echo "$s3";

echo "$s3" |  tr -s ' ' | cut -d ' ' -f 4;
echo "$s3" |  tr -s ' ' | cut -d ' ' -f 5;

Now I need to store the result from the echo's at the same variable like these:

41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz

So I could compare the results from s3cmd ls --list-md5sum with my md5sum *.gz running locally.

update Using the diff approach my command looks like:

ls --list-md5 s3://company-backup/company/amazon/ip-11-11-111-11/backup/server245/ \
| tr -s ' ' \
| cut -d ' ' -f 4,5 \
| sed 's= .*/= =' \
| diff -w - <(md5sum /backup/server245/*.gz)

Giving the follow result:

< d41d8cd98f00b204e9800998ecf8427e 
< 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
< 541b1bf78682f48867cc99dbb53c4c3a app.20121118040001.gz
> 41eae9b40d23de2f02bf07635870f6d0  /backup/diveo245/app.20121117040001.gz
> 541b1bf78682f48867cc99dbb53c4c3a  /backup/diveo245/app.20121118040001.gz

But how to know the file that is different then ? ('cause all that are equal I can delete alredy)

share|improve this question
can't you just Var=$(echo....)? – demure May 22 '13 at 13:51
up vote 5 down vote accepted

You need no variables. You can use diff to compare the outputs of two different commands:

s3cmd ls --list-md5 \
    | tr -s ' ' \
    | cut -d ' ' -f 5,6 \
    | sed 's= .*/= =' \
    | diff -w - <(md5sum *.gz)

You might need to sort the outputs to have the files in the same order.

share|improve this answer
Or omit the diff part, write the output to a file like s3.md5 and use md5sum -c s3.md5 to check against your local files. – mpy May 22 '13 at 14:06
I just don't understand the output of the diff and how to handle with it. I update my post with the output. – Valter Henrique May 22 '13 at 15:03
@ValterHenrique: In your original question, you were running md5sum *.gz. If you include path to the md5sum arguments, you have to delete it from its output, or cd /backup/server245/*.gz and use the command I provided. – choroba May 22 '13 at 15:05
got it, thank you for you help friend! It really help me out. – Valter Henrique May 22 '13 at 15:10

@choroba's answer is the way to go, but here is another way using variables as requested:

s3=`s3cmd ls --list-md5 \
 s3://company-backup/company/amazon/ip-11-11-111-11/backup_server245/ | 
 awk '{print $4,$5}' | awk -F"[/ ]" '{print $1,$NF}'`
echo $s3
 41eae9b40d23de2f02bf07635870f6d0 app.20121117040001.gz
share|improve this answer
+1 for the awk -F ... $NF trick. – mpy May 22 '13 at 15:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.