I have quite a large text file (around 20GB) that I use as a simple database, so each record is separated by a new line, and the breaking of this format will cause problems. This file also contains happens to contain some NUL characters, or at least that's what I suspect as grep treats it as a binary file.

I've come across this question and answer, which states:

Some reading has indicated that grep looks for a null character in the first thousand or so bytes, then determines from that whether or not a file is 'binary'.

For this reason, I'm thinking of stripping these characters from the file with something like:

tr < file-with-nulls -d '\000' > file-without-nulls

But I want to be sure that doing so won't break the formatting of the file. Is this a possibility at all?

  • 1
    What about just doing it and saving to a new file. Then look at whether the new file works properly. Text files don't typically contain nulls, so we have no idea what function they might be serving. – fixer1234 Sep 19 '18 at 3:02
  • 2
    Are the null character coming from mixing UTF-16 with UTF-8 ? UTF-16 text contains nulls. – matzeri Sep 19 '18 at 6:48
  • 1
    data could mean that different test coding were mixed together. If the null are coming from normal ASCII characters in UTF-16 rappresentation you should see alternate null/ASCII couples. In that case removing the null is just converting the UTF-16 to ASCII. However if you have UTF-16 code that exceeds the ASCII range, removing the null will just leave the other characters in the wrong encoding. – matzeri Sep 19 '18 at 19:29
  • 1
    If it's plain ASCII text, it shouldn't contain anything but text characters and LF or CR/LF. There is no formatting other than line breaks. If the content contained an extended character set, I don't think stripping out nulls would change anything, as matzeri already suggested (I don't think null changes the character). However, you don't need to verify the effect of every last null. If they came from something like UTF-16, all of them will have the same effect if removed. Find the location of a few examples and verify those after cleanup. (cont'd) – fixer1234 Sep 19 '18 at 20:23
  • 1
    That said, if it is absolutely critical that you not accidentally modify the data, don't remove the nulls. They apparently aren't a source of problems, and 1800 in a 20GB file won't make a real difference. – fixer1234 Sep 19 '18 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.