Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a tool available for Windows (command line, gui, script, etc.) that can recurse a directory and identify all files encoded as UTF-16?

share|improve this question
    
Generally speaking there is no way to automatically and without error detect the encoding of a text file. Having said that: if the content is actually just characters from the ASCII range (or mostly from that range) then checking for files where every second byte is 0 is a good start. –  Joachim Sauer May 4 '11 at 15:37
    
@Joachim: I think for large enough files the detection errors should be negligible. Something like "Bush hid the facts" becomes exponentially impossible once the character count is large enough. –  Philipp May 4 '11 at 16:10
    
@Philipp: I didn't even know of this particular case. Thanks! But the amount of errors to expect depends a lot on the actual content of your files: if it's all basically english ASCII text, then the detection rate will be pretty good (perfect or near perfect, I'd guess). But if you have UTF-16 encoded Chinese, Arabic, Swahili and Hindu texts in addition to lots of binary data, then it will be much worse. –  Joachim Sauer May 4 '11 at 16:16
1  
@Joachim: Agreed. From my experience, UTF-16 files without BOM are often generated by Windows system tools (installer scripts, maybe the registry editor) because such a file is essentially a memory dump of an UTF-16 string. Such files often contain lots of ASCII markup that makes them simple to identify. The other way round (deciding whether a valid UTF-16 file is in fact UTF-16) is much harder, of course. Maybe one could test whether large portions of the file belong to a single script, whether UTF-16 CRLF sequences occur, etc. –  Philipp May 4 '11 at 17:11
add comment

migrated from stackoverflow.com May 5 '11 at 6:14

This question came from our site for professional and enthusiast programmers.

2 Answers

It would not be hard to make one, read the first two bytes of every file and see if they are set to FF FE (windows) respectively.

share|improve this answer
add comment

A slow way would be to take any conversion utility and run it against all files in a directory. Those files converted successfully from UTF-16 to another format are most likely the ones you need. For that task you can pick an available tool like Character Set Converter.

Or you can write such tool using C++ code snippet from this article Conversion between Unicode UTF-16 and UTF-8 in C++/Win32. Custom tool may be optimized to give up on first conversion error and not saving converted buffer into a file.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.