Is there a tool available for Windows (command line, gui, script, etc.) that can recurse a directory and identify all files encoded as UTF-16?
4 Answers
This tool allows you to detect the file encoding type given standard information such as search pattern and file path:
File Encoding Checker is a GUI tool that allows you to validate the text encoding of one or more files. The tool can display the encoding for all selected files, or only the files that do not have the encodings you specify.
I have not used it myself, so you may want to check it out.
-
2The link is dead, but it seems to be available at github.com/amrali-eg/EncodingChecker– VictorAug 30, 2022 at 16:47
A slow way would be to take any conversion utility and run it against all files in a directory. Those files converted successfully from UTF-16 to another format are most likely the ones you need. For that task you can pick an available tool like Character Set Converter.
Or you can write such tool using C++ code snippet from this article Conversion between Unicode UTF-16 and UTF-8 in C++/Win32. Custom tool may be optimized to give up on first conversion error and not saving converted buffer into a file.
For UTF-16 files with BOM - PowerShell command
gci . -Include *.txt -Recurse | `
% { $c = gc $_.FullName -TotalCount 2 -Encoding Byte; `
if ( $c.Length -gt 0 -and `
(($c[0] -eq 255 -and $c[1] -eq 254) -or `
($c[0] -eq 254 -and $c[1] -eq 255)) `
) {$_.FullName} `
}
It would not be hard to make one, read the first two bytes of every file and see if they are set to FF FE (windows) respectively.
0
is a good start.