2

I am working with a plain text file that has invisible characters that I do not recognise. How can I identify them?

In Atom, they show as blanks when I toggle to show invisible characters. They do not show as a common space (the one that Atom shows as a small centered dot).

In BBEdit, it shows as a centered dot that looks slightly thicker than the common space. Replacing non-ASCII characters (with 'zap gremlins') does not replace it.

I can copy the character into a regular expression, and the query will find the character. It is not recognised as a white space character with \s.

I will copy the character here (between the arrows), but I have no idea if it actually shows up! -> <-

(wow, pasting an unknown invisible character felt absurdly awkward...)

4

Using a hex editor should reveal the hex codes you could then look up or search for.

If you wanted to stick with a (bash?) terminal, you could put the whole file through hexdump / hd, or maybe grep an offending line and just pipe it to hd so you're only looking at one line, similar to:

grep "unique line text" file | hd

Or get only the Nth line with
sed 'Nq;d file'

There's also the regular expression character class for all printable characters:

‘[:print:]’
 Printable characters: ‘[:alnum:]’, ‘[:punct:]’, and space.

Searching for the inverse (-v) of those might be useful, like
grep -v "[[:print:]]"

Or if you can copy it successfully, you could just paste it into a hex editor, or an echo " " | hd command...

  • 1
    Thanks! Apparently it was a non-breaking space (c2 a0). In my regex \u00A0 selects the character. – Matthijs Feb 12 at 13:10

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.