31

What is the sed incantation to remove null bytes from a file? I'm trying:

s/\000//g

but that is stripping out strings of zeroes.

s/\x00//g

seems to have no effect. I'm trying to do this in a sed script, so I'm not sure the echo trick will work.

33

I don't know how you can exactly achieve this with sed, but this is a solution that works with tr:

tr < file-with-nulls -d '\000' > file-without-nulls

This is a solution for sed that works in some occasions but not all:

sed 's/\x0//g' file1 > file2

This is a solution that involves replacing into space characters which should work in all occasions:

sed 's/\x0/ /g' file1 > file2
  • 8
    that looks like a very incomplete answer. why would it work on some occasions and not others, and if so then wouldn't an example be useful? – barlop May 24 '11 at 19:11
  • @barlop: Because of the way it is implemented? The OP didn't specify one and I'm not going to enumerate every single implementation... – Tom Wijsman May 24 '11 at 20:50
  • 3
    Well that then sounds fine to me, so you're saying it depends on the implementation of SED. If you hadn't said that you'd have left open the possible suggestion that one implementation of SED may remove nulls from one file and not another file, depending on the data in the file. – barlop May 24 '11 at 22:03
  • 2
    shouldn't it be "tr -d '\000' < file-with-nulls > file-without-nulls" ? – Seamus Abshere Jul 12 '13 at 19:32
  • 1
    Works for me™. Also useful: -i parameter to convert file in place. – zbyszek Apr 19 '16 at 19:20
8

tr tripped over some other bytes in my file and sed didn't replace anything. I ended up doing it not in sed but in Python:

f = open('file-with-0bytes.dump')
for l in f.readlines():
  print l.replace("\0", '')

Here's a pipeable one-liner:

python -c 'import sys; sys.stdout.write(sys.stdin.read().replace("\0", ""))'

I also noticed some commands actually leave the null bytes there but they're not visible anymore, at least not in an OSX terminal. I used hexdump to debug this.

3

It's quite easy to use Perl to perform a regex. Just replace sed with perl -np -e:

$ printf 'one\0two\0' | perl -np -e 's/\0/\n/g'
one
two

With the -n option, regexes are run line by line, just like sed.

If you want to use zero bytes as record separators, use Perl's -0 option.

$ printf 'one\0two\0' | perl -np0 -e 's/^/prefix /; s/\0/\n/g'
prefix one
prefix two
$ printf 'one\0two\0' | perl -np -e 's/^/prefix /; s/\0/\n/g'
prefix one
two

You can look up the command-line options of Perl by running perldoc perlrun.

1

To match a null byte, I use this regex with Cygwin's SED:

[^\x01-\x7F]

  • This is the answer to the question for sed on GNUWin32. It actually strips out more than just nulls. It may or may not work for you depending on what you want to match, and the implementation of sed. – Itsme2003 Apr 27 '18 at 2:54

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.