Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I doing a nrpe check for my nagios about the files that I backup, sending to Amazon S3. I have two files then, s3, which list all my files that are in Amazon already. And local which list all my files in my machine. I need to know what is in local and not is in s3. Example:


2013-07-01 04:49       145   32ddd73694ab610c3f077a88fb9f3300  s3://company-backup/company-logs/file.1.gz
2013-07-01 04:57       145   68adf73694ab610c3f077a88fb9f3300  s3://company-backup/company-logs/file.2.gz
2013-07-01 05:12       145   36ede43644ab710c8f077a88fb9f3300  s3://company-backup/company-logs/file.3.gz


32ddd73694ab610c3f077a88fb9f3300  /company-logs/file.1.gz
68adf73694ab610c3f077a88fb9f3300  /company-logs/file.2.gz
36ede43644ab710c8f077a88fb9f3300  /company-logs/file.3.gz
72ede57455eb850c8f078a88fb9f5900  /company-logs/file.4.gz

Notice that 72ede57455eb850c8f078a88fb9f5900 /company-logs/file.4.gz isn't in s3 file. And this is what should return to me, how can I do that ?

Thank you!

share|improve this question
up vote 2 down vote accepted

A one liner:

diff <(awk '{print $4}' s3 | sort) <(awk '{print $1}' local | sort)

Basically, what it does is compare the output of awk, which we'll use to print the fourth and first column of the files, respectively. We need to sort the output in case the hashes are in different orders. The <(…) syntax is process substitution and allows us to run diff on the output of two commands, while usually it'd only allow you to work with files.

It outputs:

> 72ede57455eb850c8f078a88fb9f5900

So you'll know 72ede57455eb850c8f078a88fb9f5900 is only in the "right" file, i.e. local.

share|improve this answer
That's some slick awk magic. :) – Justin Pearce Aug 12 '13 at 18:57
Heh, thanks. It beats cut in those cases where multiple spaces and not tabs (or other single delimiters) are used to separate columns (at least in terms of readability). – slhck Aug 12 '13 at 19:11
awesome! thanks! – Valter Henrique Aug 13 '13 at 13:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.