Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have an executable that generates a text file as its output. The problem is that the text file comes out with a binary file flag of some sort. The result is something like this:

$ grep "grep string" output_file.txt
Binary file output_file.txt matches.

$ grep -a "grep string" output_file.txt
[correct results]

Some reading has indicated that grep looks for a null character in the first thousand or so bytes, then determines from that whether or not a file is 'binary', so my question is two-fold:

  1. Is there an easy way to strip null characters from my files (I can do this as part of my post-processing) to ensure that grep works correctly without the -a flag?

  2. Is there something obvious I should look for in my code to prevent null characters from being written to the file? I've looked through the code quite thoroughly and I don't see any obvious culprits.

    .

share|improve this question

2 Answers 2

up vote 3 down vote accepted

I can answer at least the first question. If you're using Unix/Linux you can use tr

tr -d '\000' < filein > fileout

where \000 is the null char. You can also strip all non-printable chars as you can see on the example here: "Unix Text Editing: sed, tr, cut, od, awk"

Regarding your second question, I don't know which is your programming language but I'd search for uninitialized variables which could be end being printed to the output file.

share|improve this answer
    
I would vote this up if I could, but I'm apparently too new. :-/ –  brightwellcd Aug 18 '11 at 16:53
    
I found a null string in my output. I ran this tr script and did a visual diff; quickly found the problem. I'll upvote this if/when I get enough reputation to do so. Thanks. –  brightwellcd Aug 18 '11 at 17:58

I'm going to make a guess....

Your program writes the file in UTF-16, an encoding of Unicode that uses two bytes for each character. Every second byte is, most of the time, a null.

iconv -f utf-16 -t utf-8 < filein > fileout

will convert it to UTF-8, which most coreutils are comfortable with.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.