8

I have an executable that generates a text file as its output. The problem is that the text file comes out with a binary file flag of some sort. The result is something like this:

$ grep "grep string" output_file.txt
Binary file output_file.txt matches.

$ grep -a "grep string" output_file.txt
[correct results]

Some reading has indicated that grep looks for a null character in the first thousand or so bytes, then determines from that whether or not a file is 'binary', so my question is two-fold:

  1. Is there an easy way to strip null characters from my files (I can do this as part of my post-processing) to ensure that grep works correctly without the -a flag?

  2. Is there something obvious I should look for in my code to prevent null characters from being written to the file? I've looked through the code quite thoroughly and I don't see any obvious culprits.

    .

3 Answers 3

13

I can answer at least the first question. If you're using Unix/Linux you can use tr

tr -d '\000' < filein > fileout

where \000 is the null char. You can also strip all non-printable chars as you can see on the example here: "Unix Text Editing: sed, tr, cut, od, awk"

Regarding your second question, I don't know which is your programming language but I'd search for uninitialized variables which could be end being printed to the output file.

2
  • I would vote this up if I could, but I'm apparently too new. :-/ Aug 18, 2011 at 16:53
  • I found a null string in my output. I ran this tr script and did a visual diff; quickly found the problem. I'll upvote this if/when I get enough reputation to do so. Thanks. Aug 18, 2011 at 17:58
6

I'm going to make a guess....

Your program writes the file in UTF-16, an encoding of Unicode that uses two bytes for each character. Every second byte is, most of the time, a null.

iconv -f utf-16 -t utf-8 < filein > fileout

will convert it to UTF-8, which most coreutils are comfortable with.

6
  • Interesting, and I didn't know this about UTF-16. A question about this command - what exactly does the conversation remove from or do to the file? How will it behave in the use case of my question here? Oct 10, 2018 at 21:37
  • 1
    @Hashim: It doesn't quite remove anything; it reads values in one representation and writes the same values out in another. (Much like converting between hex and octal, or between PNG and BMP.) UTF-16 represents each codepoint value as a fixed-length two-byte code (or a pair of two such codes), which naturally has to be padded with a 0x00 byte if the value is below 256, while UTF-8 represents the same value as a variable-length code which doesn't require null-padding. How it'll behave with your file depends on whether your file is UTF-16 to begin with. Oct 10, 2018 at 21:56
  • @Hashim Is there a way of determining whether a file is UTF-16? Doing file myfile.txt simply shows the file as data. Oct 10, 2018 at 21:58
  • If it's text and looks like text in your text editor, look at what encoding the editor has detected. Try to perform the conversion, and check if the result still looks like text in your text editor. Or do a hexdump of your file, if you see that "every second byte" is 0x00, that almost always means UTF-16. Oct 11, 2018 at 4:38
  • Unfortunately opening the file in an editor is out of the question as the files I'm working on are too large - all more than 10GB. If there are no NUL bytes in the second column of a hexdump is it safe to conclude that the file is definitely not UTF-16? Oct 11, 2018 at 18:05
0

Another option could be using strings tool, which is actually even more powerful than just to "fix" encoding or null chars, it can fetch text data from any real binary, so

$ strings output_file.txt | grep "grep string"

usually quite handy to quickly find something in binary file, e.g.

$ strings firmware.bin | grep VERSION

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .