display and remove all chars > ascii code 127

Question

How can I display and remove all chars > ascii code 127 from a file?

UPDATE Dennis has solved it but there is some interesting discussion about number ranges.

I have a regex for it and this is some commentary on how I worked it out, with a few tests.

0080-00FF     00[89A-F][0-9A-F]
0100-0FFF     0[1-9A-F][0-9A-F]{2}
1000-FFFF     [1-9A-F][0-9A-F]{3}

And the tests-

$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "[0-9A-F]{4}" | wc -l
85

$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "00[0-7][0-F]" | wc -
l
72

$ xxd -p -u a.aa.txt | sed -r "s/[0-9A-F]{4}/\0 /g" | grep -oP '((?!00[0-7][0-9A-F])(?=[^ ]).){4}' | wc -l
13


$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "00[89A-F][0-9A-F]|0[1-9A-F][0-9A-F]{2}|[1-9A-F][0-9A-F]{3}" | wc –l
13

I know sed doesn't have negative lookahead, but we have the positive version. And also it helped a bit as a test.

Answer 1

One approach to take would be to convert the file to hex digits, remove the digit patterns that you don't want, then convert back.

$ echo 'A Unicode character: [ñ]' | xxd -p | sed 's/c3b1//' | xxd -r -p
A Unicode character: []

You could use AWK or any other text manipulation technique in place of sed. Be careful of ambiguous sequences.

Let me know if this approaches what you have in mind.