0

I frequently use this command:

LC_ALL=C.UTF-8 egrep -laxv '.*' filename

this tells me if the file contains any non-UTF8 characters (actually I usually use this in conjunction with find to scan many files at once)

I can remove the -l to get the actual line(s) that contain UTF-8 characters instead of just the filenames and that's usually good enough, I can usually look at the line and spot the problem character

however I'm currently dealing with a file with very long lines and I haven't yet been able to locate the offending character just by eyeballing it

I'd like to modify the grep command to print out just the non-UTF8 character instead of the entire line

Unfortunately -o doesn't help because it's a -v negative match

I don't want to delete the character (yet) I just want to figure out which character it is

I tried something like ```LC_ALL=C.UTF-8 egrep -ao '[^.]' but "." inside a character group is treated literally so it doesn't do anything but output every character of the file that isn't a "."

If I were searching for non-ASCII characters I know I could use the [[:ASCII:]] character class but there doesn't seem to be an equivalent for UTF-8

I tried searching for '[^[[:print:]]]' in the problem file but it failed to find anything

I tried alternate methods such as running the file through UTF-8 converters and comparing them to the original file, but they all claim that the file is already fully UTF-8. I think I might be dealing with a bug in grep that causes a valid UTF-8 character to be detected as invalid, however, to investigate further, I'd need to know what character is the actual issue

When dealing with this with another file earlier, I did some trial and error and determined that (for that specific other file) it was the Korean character 획 that was the source of the problem. For the file I'm dealing with now, there is a bunch of Korean in it, however, there are no instances of 획 so it must be a different character causing the problem this time. The file I was dealing with earlier only had 4 Korean characters in it so it was easy to figure out which one was the cause of the problem but the file I'm dealing with no has a lot more and I don't really want to do this by trial and error

4
  • Have you tried highlighting with --color=always? Mar 27, 2022 at 23:56
  • Related: stackoverflow.com/questions/3001177/… Mar 28, 2022 at 0:26
  • if your sed uses the same regex code as your grep, sed /^.*// . Or, fold to a more convenient linesize; could first cat -n to more easily match back to input. Mar 28, 2022 at 1:54
  • Can you get the whole line printed, and then remove ASCII characters from it? Mar 28, 2022 at 5:13

0

You must log in to answer this question.

Browse other questions tagged .