I want to use a Unix shell command to find all UTF-16 encoded files (containing the UTF-16 Byte Order Mark (BOM)) in a directory tree. Is there a command that I can use?


Though you asked to find the BOM, using file might even give you results when such BOM is not present. From man file:

If a file does not match any of the entries in the magic file, it is examined to see if it seems to be a text file. ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets (such as those used on Macintosh and IBM PC systems), UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC character sets can be distinguished by the different ranges and sequences of bytes that constitute printable text in each set. If a file passes any of these tests, its character set is reported.

Hence, for example:

find . -type f -exec file --mime {} \; | grep "charset=utf-16"

You can use grep:

 grep -rl $(echo -ne '^\0376\0377') *

(Tested with bash and GNU grep, might work with others.)


The $(echo... part generates the BOM (Hex FE FF, as octal escape sequences), this is then fed to grep as its pattern, prepended with '^' (=match start of line).

-r is recursive search, -l makes grep print the names of files it found (instead of the matching line).

This might be a bit wasteful, as grep will scan each file completely, rather than just the start. If it's mostly small text files, it will not matter. If you have loads of files with several MB, you'll have to write a perl script :-).

Alternatively, you could try file (combined with find+xargs). file will identify UTF-16 (as "UTF-16 Unicode character data"). I don't know how reliable it is, however (as it uses heuristics).

  • 1
    "...as grep will scan each file completely..." - only the ones who do not match. -l makes grep stop at the first occurrence. With -I one also makes grep ignore binary files. May 22 '12 at 10:01
  • Can you explain what the $(....) wrapping of the echo is doing? Also - what is the * here - is it just a wildcard for file-matching, i.e. grep all files?
    – El Ronnoco
    Jul 28 '21 at 9:52
  • $(..) is a Bash feature called "command substitution" - see the bash manual. And yes, "*" is a wildcard for all files.
    – sleske
    Jul 28 '21 at 15:52
  • Slightly enhanced by adding a file filter: grep -rl --include *.h --include *.rc $(echo -ne '^\0377\0376') *. Note that I swapped the BOM bytes, because UTF-16LE start with the BOM 0xFF 0xFE.
    – Heri
    Oct 14 '21 at 11:53

Here is the script that I use to find UTF-16 files, and subsequently convert them to UTF-8. #!/bin/sh

find ./ -type f |
while read file; do
    if [ "`head -c 2 -- "$file"`" == $'\xff\xfe' ]
        echo "Problems with: $file"
        # If you want to convert to UTF-8 uncomment these lines.
        #cat "$file" | iconv -f UTF-16 -t UTF-8 > "$file.tmp"
        #mv -f "$file.tmp" "$file"

If you have it, you can use enca:

enca -L none * 2>/dev/null | grep  -i "Universal character"

Thanks for the help everybody. What worked best on my Mac was:

find . -type f -exec awk -F '\n' '/^\xFE\xFF|\xFF\xFE/ { print FILENAME; nextfile } { nextfile }' {} \;

It's based on sleske's solution, but takes into account that the Byte Order Mark can be reversed. It also uses awk to stop looking for the BOM after the first line, since the BOM must be at the beginning of the file. The \x escaping used to specify the BOM works with bash, I don't know if it works with other shells.

The enca tool suggested by ghostdog74 also will do the job, but it wasn't present on my Mac.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.