Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a tool available for Windows (command line, gui, script, etc.) that can recurse a directory and identify all files encoded as UTF-16?

share|improve this question

migrated from May 5 '11 at 6:14

This question came from our site for professional and enthusiast programmers.

Generally speaking there is no way to automatically and without error detect the encoding of a text file. Having said that: if the content is actually just characters from the ASCII range (or mostly from that range) then checking for files where every second byte is 0 is a good start. – Joachim Sauer May 4 '11 at 15:37
@Joachim: I think for large enough files the detection errors should be negligible. Something like "Bush hid the facts" becomes exponentially impossible once the character count is large enough. – Philipp May 4 '11 at 16:10
@Philipp: I didn't even know of this particular case. Thanks! But the amount of errors to expect depends a lot on the actual content of your files: if it's all basically english ASCII text, then the detection rate will be pretty good (perfect or near perfect, I'd guess). But if you have UTF-16 encoded Chinese, Arabic, Swahili and Hindu texts in addition to lots of binary data, then it will be much worse. – Joachim Sauer May 4 '11 at 16:16
@Joachim: Agreed. From my experience, UTF-16 files without BOM are often generated by Windows system tools (installer scripts, maybe the registry editor) because such a file is essentially a memory dump of an UTF-16 string. Such files often contain lots of ASCII markup that makes them simple to identify. The other way round (deciding whether a valid UTF-16 file is in fact UTF-16) is much harder, of course. Maybe one could test whether large portions of the file belong to a single script, whether UTF-16 CRLF sequences occur, etc. – Philipp May 4 '11 at 17:11

It would not be hard to make one, read the first two bytes of every file and see if they are set to FF FE (windows) respectively.

share|improve this answer

A slow way would be to take any conversion utility and run it against all files in a directory. Those files converted successfully from UTF-16 to another format are most likely the ones you need. For that task you can pick an available tool like Character Set Converter.

Or you can write such tool using C++ code snippet from this article Conversion between Unicode UTF-16 and UTF-8 in C++/Win32. Custom tool may be optimized to give up on first conversion error and not saving converted buffer into a file.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.