2

I have a huge log file (several GBs), but somehow there is binary data (grep is annoyed by it) in there, which of course shouldn't be.

I know how I can read the file anyway.

I don't know how I can find the bad binary data in it so I maybe can pin point where it's logged looking at the text around it?

3
  • Well actually a text file is also a binary lol. I guess you will be looking for something that deletes all alphanumeric characters from your file, only leaving the other data in the file.
    – xdevs23
    Sep 7 '16 at 8:37
  • @xdevs23 Yeah it is :D. But what I'm looking for is something like grep for those files to see the the normal log text around it. So I can maybe blame what is adding those binary data.
    – keiki
    Sep 7 '16 at 8:56
  • I have no idea how you are going to be able to solve that. Maybe someone else knows.
    – xdevs23
    Sep 7 '16 at 9:13
1

I've just hit the exact same problem (although it's only a multi-megabyte log file). As with a lot of problems, it just takes a couple of commands together.

cmp /path/to/file.log <(strings /path/to/file.log)

cmp compares files and tells you where they differ (unlike diff, which tells you how they differ). strings returns valid text strings from binary files. <(…) lets you treat the output of a command as a file descriptor into another command.

Basically, you compare the log file with the text strings in the log file so that you find where they first differ.

For example, I get A and B differ: byte 1450315, line 6390. Running tail -n +6390 /path/to/file.log | less shows the log starting at the "bad" line, or you can pipe through | hexdump -C | less to see the hex (piping through head -n 1 didn't work for me because the binary was \x00 characters, which only showed when there was a pager)

(Note: This may not work well with multi-gigabyte logs if the machine does not have sufficient memory - I don't know how memory efficient strings and cmp are)

0

I have solved the same problem. What worked for me, was to simply head the file incrementally and grep it to see at which line the binary characters occur.

At the beginning, I called head -n 1, there was no binary character. Then head -n 2, then head -n 3... and so on. Soon I found the line, where the binary character were present.

0

Building on @ibboard's great idea: find-non-printable.sh:

#!/bin/sh
usage="$0 FILE - Locates first non-printable byte, as in 'FILE - differ: byte 21881, line 507'";
n_bytes=$(stat --printf='%s' "$1");
# -w: --include-all-whitespace
strings -w "$1" | cmp -n "$n_bytes" "$1";
  • No output when no non-printable characters are found, instead of message like cmp: EOF on FILE after byte 1677, line 47.
  • Simple pipe instead of using <(). Works with a POSIX shell instead of requiring Bash or ZSH. Output shows - as file name instead of /dev/....
  • Works as expected when the file contains blank lines. This requires a strings implementation that supports -w, such as GNU strings. Without this, the result just points to the first blank line, which is often not the desired result.
  • User needs to type file name only once.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.