I'm trying to replicate the behavior of the sha1sum executable in some java code, however, in the process I've discovered that sha1sum appears to behave differently given the same input in two scenarios.

Assume input of '12345' without the single quotes and with no newline.

If I put this data into a file (file1) and run sha1sum from the command line:

$ sha1sum file1
8cb2237d0679ca88db6464eac60da96345513964  file1

However, if I do this, I get a different result:

$ cat file1 | grep -vi foo | grep -vi bar | sha1sum
2672275fe0c456fb671e4f417fb2f9892c7573ba  -

Using the apache commons-codec jar, I'm able to read in file1, get it's contents, and perform a .shahex() on the content and get the first result. However, I need to get the second result (due to legacy code) and I can't figure out why sha1sum is behaving differently, or what grep is doing to the input.

The system is running CentOS 5.4 with sha1sum 5.97

Any pointers?

  • i never recognized this, nice question – matthias krull Jun 23 '10 at 15:33

grep adds a newline.

$ hd file1
00000000  31 32 33 34 35                                    |12345|
$ grep -vi test <file1 | hd
00000000  31 32 33 34 35 0a                                 |12345.|

To get the same result you should add a single \n at end of input if it is not present.

| improve this answer | |
  • After taking the advice below and adding an extra newline, I think I have things working. Thanks! – twilbrand Jun 23 '10 at 14:56

Confirm the sequences are identical, first:

$ cat file1 | grep -vi foo | grep -vi bar > /tmp/junk
$ cmp file1 /tmp/junk

otherwise you'll just be chasing your tail.

| improve this answer | |
  • This was brilliant advice, I had been printing out the results, but putting them into a file showed differences I didn't know about. Unfortunately the answer above about the newline is still correct and ends up being the ultimate solution, so I can't mark your answer correct. I would if I could mark two correct. – twilbrand Jun 23 '10 at 14:56
  • I upvoted whitequark, too, no worries. – msw Jun 23 '10 at 16:42

Is your grep aliased to have --color? Have you checked the output of your pipeline without sha1sum in it?

| improve this answer | |
  • 3
    grep only adds color (even with a --color switch) when writing to tty, i.e. when isatty(1) == 1. See man 3 isatty. – whitequark Jun 22 '10 at 22:43
  • 1
    If it were aliased to --color=always for some reason, it ignores the tty check. Otherwise you're correct. – Daenyth Jun 23 '10 at 1:20
  • nope, it is not aliased – twilbrand Jun 23 '10 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.