Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How can I display and remove all chars > ascii code 127 from a file?

file is unicode 16bit

UPDATE Dennis has solved it but there is some interesting discussion about number ranges.

share|improve this question
Why do you need that? – grawity Jan 30 '11 at 9:19
@gravvity why do you ask? – barlop Jan 30 '11 at 10:44
Because it makes a difference in how you go about it. An additional question: Do you have Unix-style tools available through Cygwin or GNUWin32 or similar? – Dennis Williamson Jan 30 '11 at 11:27
@Dennis Williamson I'm open to many options of going about it. I have both Cygwin and Gnuwin32, it'd be nice to do it via those but a problem with that is my file is unicode 16bit. I notice that the line: cat | od -c interprets \0s between each char, and grep doesn't work on it cos the letters abc as seen by grep are not contiguous for presumably the same reason. they assume 8-bit per char. – barlop Jan 30 '11 at 12:01
Since you have a Unicode file, you should know that's the real reason grawity and I asked for additional information. Removing bytes with byte values (notice that I didn't say "characters" and "ASCII codes") from a Unicode file will give weird results. What is it that you're really trying to do? – Dennis Williamson Jan 30 '11 at 15:18
up vote 1 down vote accepted

One approach to take would be to convert the file to hex digits, remove the digit patterns that you don't want, then convert back.

$ echo 'A Unicode character: [ñ]' | xxd -p | sed 's/c3b1//' | xxd -r -p
A Unicode character: []

You could use AWK or any other text manipulation technique in place of sed. Be careful of ambiguous sequences.

Let me know if this approaches what you have in mind.

share|improve this answer
yep, that is approaching what I have in mind – barlop Jan 30 '11 at 23:07
from there I can see how i'd probably do it.. od -x myfile | cut to get past FEFF at the start of it , capture pairs in the regex within my hex range, in sed. – barlop Jan 30 '11 at 23:12
actually, my cut idea wouldn't cut it.. but yes the idea of a program or set that could let me see the hex, work with regex on the hex and write it back is one general approach I had in mind.. i'd be interested to see how you tweak it to the requirements. You are welcome to post other sketches of approaches if anything springs to mind..if it's not too much trouble. i'd be interested and I think it'd be useful. – barlop Jan 30 '11 at 23:34
i'm actually getting there via that sketch you gave, and using cut , i missed a tr -d '\n' before it.. – barlop Jan 31 '11 at 0:13
actually i'm stuck trying to tweak your sketch . your sketch is fine (of course as you know!) but my tweaking is the problem. 'cos i know that after | tr -d '\n' and then cut 5-, which works, I don't want to just scan for say FEFF or [8-F][0-F][0-F][0-F] because I don't want to overlap bytes from 2 chars. And whatever I put in the find section of sed s goes. So if I try to match every byte pair then it'd delete the lot. and i'm not sure how or if it's possible with sed to replace byte pairs conditionally. – barlop Jan 31 '11 at 0:51

Looking back at this question

It looks I used one of these lines in the end

(seems to be dennis's regex)

$ xxd -p a.q | tr -d '\n' | sed "s/([0-9a-f]\{4\})/\1 /g" | sed -r "s/(00[8-9a-f][0-9a-f]|[0-9a-f][1-9a-f][0-9a-f][0-9a-f]|[1-9a-f]0[0-9a-f][0-9a-f])//g" | tr -d ' ' | sed "s/(.*)/feff\1/" | xxd -r -p >a.q2

or this (seems to be my regex)

$ xxd -p a.q | tr -d '\n' | sed "s/([0-9a-f]\{4\})/\1 /g" | sed -r "s/(00[89A-F][0-9A-F]|0[1-9A-F][0-9A-F]{2}|[1-9A-F][0-9A-F]{3})//g" | tr -d ' ' | sed "s/(.*)/feff\1/" | xxd -r -p >a.q2

this is some commentary on how I worked it out, with a few tests.

0080-00FF     00[89A-F][0-9A-F]
0100-0FFF     0[1-9A-F][0-9A-F]{2}
1000-FFFF     [1-9A-F][0-9A-F]{3}

and here are some tests that shows that regex works. I had put them into my question when I solved it but I should have put them into an answer.

Total Hex Chars 85

$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "[0-9A-F]{4}" | wc -l

Total that I want, 72

$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "00[0-7][0-F]" | wc -

Total that I don't want, 13

$ xxd -p -u a.aa.txt | sed -r "s/[0-9A-F]{4}/\0 /g" | grep -oP '((?!00[0-7][0-9A-F])(?=[^ ]).){4}' | wc -l

Total that I don't want, 13

$ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "00[89A-F][0-9A-F]|0[1-9A-F][0-9A-F]{2}|[1-9A-F][0-9A-F]{3}" | wc –l
share|improve this answer
and looking at it now i'm wondering why I didn't just grep the unicode hex characters I wanted to keep as in this test $ xxd -u -p a.aa.txt | sed -r "s/.{4}/\0 /g" | grep -oE "00[0-7][0-F]" Rather than replacing the ones I didn't want to keep with nothing. (the latter amounts to a much longer regex) – barlop Aug 30 '14 at 18:13
by the way, the first xxd -p a.q line uses the regex in dennis's comment, the second uses the regex that I came up with. – barlop Aug 31 '14 at 1:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .