up vote 1 down vote favorite
share [g+] share [fb]

Is there a tool available for Windows (command line, gui, script, etc.) that can recurse a directory and identify all files encoded as UTF-16?

link|improve this question
Generally speaking there is no way to automatically and without error detect the encoding of a text file. Having said that: if the content is actually just characters from the ASCII range (or mostly from that range) then checking for files where every second byte is 0 is a good start. – Joachim Sauer May 4 '11 at 15:37
@Joachim: I think for large enough files the detection errors should be negligible. Something like "Bush hid the facts" becomes exponentially impossible once the character count is large enough. – Philipp May 4 '11 at 16:10
@Philipp: I didn't even know of this particular case. Thanks! But the amount of errors to expect depends a lot on the actual content of your files: if it's all basically english ASCII text, then the detection rate will be pretty good (perfect or near perfect, I'd guess). But if you have UTF-16 encoded Chinese, Arabic, Swahili and Hindu texts in addition to lots of binary data, then it will be much worse. – Joachim Sauer May 4 '11 at 16:16
1  
@Joachim: Agreed. From my experience, UTF-16 files without BOM are often generated by Windows system tools (installer scripts, maybe the registry editor) because such a file is essentially a memory dump of an UTF-16 string. Such files often contain lots of ASCII markup that makes them simple to identify. The other way round (deciding whether a valid UTF-16 file is in fact UTF-16) is much harder, of course. Maybe one could test whether large portions of the file belong to a single script, whether UTF-16 CRLF sequences occur, etc. – Philipp May 4 '11 at 17:11
feedback

migrated from stackoverflow.com May 5 '11 at 6:14

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 0 down vote

It would not be hard to make one, read the first two bytes of every file and see if they are set to FF FE (windows) respectively.

link|improve this answer
feedback
up vote 0 down vote

A slow way would be to take any conversion utility and run it against all files in a directory. Those files converted successfully from UTF-16 to another format are most likely the ones you need. For that task you can pick an available tool like Character Set Converter.

Or you can write such tool using C++ code snippet from this article Conversion between Unicode UTF-16 and UTF-8 in C++/Win32. Custom tool may be optimized to give up on first conversion error and not saving converted buffer into a file.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.