Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to replicate the behavior of the sha1sum executable in some java code, however, in the process I've discovered that sha1sum appears to behave differently given the same input in two scenarios.

Assume input of '12345' without the single quotes and with no newline.

If I put this data into a file (file1) and run sha1sum from the command line:

$ sha1sum file1
8cb2237d0679ca88db6464eac60da96345513964  file1

However, if I do this, I get a different result:

$ cat file1 | grep -vi foo | grep -vi bar | sha1sum
2672275fe0c456fb671e4f417fb2f9892c7573ba  -

Using the apache commons-codec jar, I'm able to read in file1, get it's contents, and perform a .shahex() on the content and get the first result. However, I need to get the second result (due to legacy code) and I can't figure out why sha1sum is behaving differently, or what grep is doing to the input.

The system is running CentOS 5.4 with sha1sum 5.97

Any pointers?

share|improve this question
    
i never recognized this, nice question –  matthias krull Jun 23 '10 at 15:33
add comment

3 Answers

up vote 14 down vote accepted

grep adds a newline.

$ hd file1
00000000  31 32 33 34 35                                    |12345|
00000005
$ grep -vi test <file1 | hd
00000000  31 32 33 34 35 0a                                 |12345.|
00000006

To get the same result you should add a single \n at end of input if it is not present.

share|improve this answer
    
After taking the advice below and adding an extra newline, I think I have things working. Thanks! –  twilbrand Jun 23 '10 at 14:56
add comment

Confirm the sequences are identical, first:

$ cat file1 | grep -vi foo | grep -vi bar > /tmp/junk
$ cmp file1 /tmp/junk

otherwise you'll just be chasing your tail.

share|improve this answer
    
This was brilliant advice, I had been printing out the results, but putting them into a file showed differences I didn't know about. Unfortunately the answer above about the newline is still correct and ends up being the ultimate solution, so I can't mark your answer correct. I would if I could mark two correct. –  twilbrand Jun 23 '10 at 14:56
    
I upvoted whitequark, too, no worries. –  msw Jun 23 '10 at 16:42
add comment

Is your grep aliased to have --color? Have you checked the output of your pipeline without sha1sum in it?

share|improve this answer
3  
grep only adds color (even with a --color switch) when writing to tty, i.e. when isatty(1) == 1. See man 3 isatty. –  whitequark Jun 22 '10 at 22:43
1  
If it were aliased to --color=always for some reason, it ignores the tty check. Otherwise you're correct. –  Daenyth Jun 23 '10 at 1:20
    
nope, it is not aliased –  twilbrand Jun 23 '10 at 14:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.