Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have to deal with a file that has lot of invisible control characters, like "right to left" or "zero width non-joiner", different spaces than the normal space and so on, and I have troubles dealing with that.

Now, I would like to somehow view all letters in a given file, letter by letter (I would like to say "left to right", but I am unfortunately dealing with right-to-left language), as unicode codepoints, using only basic bash tools (like vi, less, cat...). Is it possible somehow?

I know I can display the file in hexadecimal by hexdump, but I would have to recompute the codepoints. I really want to see the actual unicode codepoints, so I can google them and find out what's happenning.

edit: I will add that I don't want to transcode it to different encoding (because that's what I am finding out online). I have the file in UTF8 and that is fine. I just want to know the exact codepoints of all the letters.

share|improve this question
up vote 2 down vote accepted

I wrote myself a perl one-liner, that do just that, and it also prints the original character. (It expects the file from STDIN)

perl -C7 -ne 'for(split(//)){print sprintf("U+%04X", ord)." ".$_."\n"}'

However, there should be a better way than this.

share|improve this answer

I needed the code point for some common smileys, and came up with this:

echo -n "😊" |              # -n ignore trailing newline                     \
iconv -f utf8 -t utf32be |  # UTF-32 big-endian happens to be the code point \
xxd -p |                    # -p just give me the plain hex                  \
sed -r 's/^0+/0x/' |        # remove leading 0's, replace with 0x            \
xargs printf 'U+%04X\n'     # pretty print the code point

which prints


which is the code point for "SMILING FACE WITH SMILING EYES".

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .