I've inherited a big collection of web content claiming to use utf-8 encoding, yet clearly random gibberish that shows up when displaying the pages says they are some sort of windows encoding. I've used the command iconv -f Windows-1252 -t utf-8 *19* -o test.htm to change one file, and now I see valid characters that make sense, but in front of each of those characters there is now a 'Â' uppercase A with a circumflex. Examining the binary data shows that all the special characters are hex pairs such as C292 or C297. Is there some windows charset that uses C2 as an escape and I want something different than Windows-1252 in the iconv command?
1 Answer
I have a feeling the files were produced by someone using utf-8 escape sequences to embed Windows characters as though that made sense.
Doing the above iconv followed by a sed command to simply eliminate the 0xc2 characters left in the files after conversion created files which seem to display correctly now.
I did scan for legitimate instances of 'Â' which might be present, but I'm pretty sure there weren't any accented characters at all.
-
-
Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.– Community BotJan 28 at 20:55
filecommand, at least under unix, can sometimes show the charset, as an heuristic. Also "now I see": how do you look at the file? which tool?<meta>tags. Does any of them actually declare the UTF-8 encoding? (I don’t understand why this was migrated off Webmasters, it should have been perfectly on-topic there.)C2 92in the source or in the output?