Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Our users are experiencing a very discouraging issue in regards to how MS Word (in Windows) handles non-unicode characters. This issue is confirmed in both Word 2007 and the Word 2010 Beta using Windows XP SP3; I suspect it works the same way in 2003.

Issue:

  1. A user creates a document using a non-unicode font, entering characters to represent scientific notations. For example, he enters a Mu (µ). Note: I pasted in a unicode-compliant Mu for reference.
  2. The user opens his document and attempts to copy / paste this non-unicode character representing a Mu into a web browser for entry into our system. It pastes as an unrecognized character. This is expected.
  3. The user opens his document, selects the non-unicode character and adjusts its font to "Arial Unicode MS," saving the document. He closes / re-opens the document for good measure. Once re-opened, he copies what should be a unicode Mu and pastes it into the web browser. It is still represented as an unrecognized character.
  4. The user creates a new document, sets the font to "Arial Unciode MS" and creates a Mu. He copies this Mu into the web browser and it pastes over in Unicode, as expected.

Conclusion:

Word is not actually converting non-unicode characters into unicode characters when it should, when a unicode font is selected. Instead, it is taking a best-guess for display reasons but doing no actual conversion.

How do I overcome this problem?

  • Can I change some setting in Word to force a conversion? Preferable.
  • Is there a "cleaner" app or Word macro that will do this?
  • Other solutions?

Additional Notes:

  • Re-typing the affected documents using unicode is not an option
  • This is not an issue in Mac OS X using the most recent version of Word. A sample case such as in (3) results in a unicode Mu being pasted into the browser.

Please help!

share|improve this question

migrated from stackoverflow.com Jul 13 '10 at 18:37

This question came from our site for professional and enthusiast programmers.

    
StackOverflow is a forum for programming-related questions. Try SuperUser.com instead. –  Borealid Jul 13 '10 at 18:20
1  
Noted should be that you don't need to create a new question over there. With enough votes this question will be moved sooner or later. Just have patience. –  user26750 Jul 13 '10 at 18:24
    
What are non-unicode characters? –  Philipp Jul 13 '10 at 18:26
    
@Philipp: That's generally the layman's term for characters outside ISO-8859-X range (or whatever default encoding the underlying platform is using, e.g. CP-1252 at Windows or Roman at Mac OS). Very contradicting term indeed since Unicode actually covers every character the human linguistic world is aware of ;) –  user26750 Jul 13 '10 at 18:27
    
Cut and Paste issues can be related to programming, even if this specific question isn't. Substitute "my program" for "MS Word" and the topic doesn't really change. –  Mark Ransom Jul 13 '10 at 18:33

3 Answers 3

A lengthy process but I normally convert such files into images and then process those images through any OCR software. That helps. But, I was myself searching for an even better option.

share|improve this answer

Thanks Mark. You're right that Paste Special won't work for non-unicode text. Do you know of a utility that will "clean" non-unicode documents. For example, load document that contains non-unicode Mu symbol and convert said Mu symbols into unicode-compliant Mus? Is there a macro or plugin for Word that will do this?

Best regards, Kirby

share|improve this answer
    
I'm sorry that this isn't an answer... I'm not sure why it won't let me comment to answer your questions, Mark. 1) Word created the original document with "symbol" Mus (e.g., not unicode). I've come to find there is a term for this: symbol-abuse. 2) When pasting from Word into the browser it carries over as an unrecognized character (e.g., a white empty box) 3) When converting the symbol Mu to Arial Unicode, it displays a Mu. The problem is that it's just translating but not converting. For example, if you take a symbol Mu and change its font to Arial Unicode, it's still a symbol Mu. –  Kirby Jul 19 '10 at 21:31
    
Could someone please tell me how I can reliably (and programatically) convert a document that uses symbols to be unicode? –  Kirby Jul 19 '10 at 21:32

Try using Paste Special; there should be an option for Unicode text.

Note that if the source document was created with a Symbol font, this won't help. Windows doesn't really know that the character is related to a specific Unicode character, the symbol fonts were created before Unicode as a way of meeting a need and the two aren't interchangeable.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.