Is it safe (no data loss) to convert ANSI to UTF-8 and then back to ANSI?

I have read that you can lose data going from UTF-8 to ANSI.

But if the file was changed from ANSI to UTF-8 (and not changed further while in UTF-8) and then changed back to ANSI, is this 100% safe?

  • If you are really worried you can compare the files after to see if they're identical, then you know if it's 100% There may be a command to do a byte by byte check. The fc command might do a byte by byte check – barlop Feb 14 '19 at 16:37

It's probably safe, but only if you convert between the same encodings both times.

UTF-8 by itself isn't a character set – it is a way to encode Unicode into bytes. It can represent the same characters as UTF-16, the encoding that modern Windows uses. So the real question is whether converting to Unicode can lose information – and AFAIK, the answer is "it shouldn't, but it sometimes might":

  • The Old New Thing has a footnote about this:

    Bonus chatter: Even the round trip from ANSI to Unicode and back to ANSI can be lossy, depending on the flags you pass regarding use of precomposed characters, for example.

    Unicode has several canonical forms – for example, ã can be stored both as a single codepoint (precomposed), or as plain a + combining tilde (decomposed). Windows prefers the former, macOS prefers the latter.

  • I'm not entirely sure whether e.g. Windows-932 counts as "ANSI", but I wouldn't be surprised if there were issues (as mentioned on Wikipedia) due to the same byte doubling as both a ¥ symbol and a path separator that's normally a backslash...

Meanwhile, there is no encoding or codepage called "ANSI". It's the name of a standards organization which has defined several text encodings. Within Windows, the term means a large set of "Windows-125x" encodings for various countries and languages (somewhat corresponding to ISO 8859 encodings, and allegedly based on early drafts written by ANSI).

So it is very possible that one system calls Windows-1251 "ANSI" and another uses Windows-1257 for the same, and as a result, each can represent characters that the other cannot. (In fact, latest Windows 10.1809 even allows UTF-8 to be the "ANSI" encoding.) In the case of differently configured systems, even if the initial conversion to Unicode doesn't lose information, converting back to "ANSI" will.

  • You write "It's probably safe, but only if you convert between the same encodings both times." <--- Do you mean it's not 100% safe even if you convert between the same encodings both times? – barlop Feb 14 '19 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.