54

When saving the text "Bush hid the facts" in notepad under Windows XP, how come when you reopen it shows squares instead of the text?

I saw it in this video if you need an example

http://www.youtube.com/watch?v=9bK9-sc_uus&feature=related

2
  • 28
    By the way, it's the same for any sequence of 4-3-3-5 letters, not just this one. Jul 31, 2009 at 8:42
  • 15
    Example: "John ate the bacon"
    – Troggy
    Aug 18, 2009 at 18:24

1 Answer 1

93

This is due to a problem with the Win32 API function IsTextUnicode dating back to Windows NT 3.5. If a file is encoded in ANSI, the function will interpret it as UTF-16LE resulting in unreadable characters.

This fascinated me too back when I discovered it since I was kind of young and naive, I thought it was an actual conspiracy :)

There is actually a Wikipedia article on this you can find here.

9
  • 17
    Interesting. +1 for the Wiki article that taught me the word "mojibake" and its particularly meta warning that "without proper rendering support, you may see question marks, boxes, or other symbols..." :-)
    – jtb
    Jul 31, 2009 at 1:33
  • 3
    +1 because, despite using Windows for as long as I can remember, I somehow never came across this! Jul 31, 2009 at 2:56
  • 7
    Well, it's clearly a bug, because the software incorrectly. The best you can argue is that bugs like this are impossible to eliminate without losing other functionality. And, heck, Microsoft fixed it in Vista [according to Wiki], so someone there obviously thought it was a bug too.
    – John Fouhy
    Jul 31, 2009 at 4:30
  • 11
    It's not a bug if it does exactly what it advertise (i.e. documented) to do. It's specified precisely that it's a statistical test and not foolproof, and the shorter the input, the higher the error rate. It just so happens that in this case, it happens with a sentence that make sense to human. This particular sentence doesn't work with Vista & 7 because the implementation of IsTextUnicode have been changed and presumingly improved and it now report correctly for this sentence. What we have is better or worse false positive / negative rate, not bugs.
    – KTC
    Jul 31, 2009 at 6:00
  • 5
    "It's not a bug if it does what it's supposed to." Yeah maybe the technical term is 'design flaw' or something, but I think most people would still say it's ok to call it a bug.
    – davr
    Nov 18, 2009 at 0:37

You must log in to answer this question.