Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to use a Unix shell command to find all UTF-16 encoded files (containing the UTF-16 Byte Order Mark (BOM)) in a directory tree. Is there a command that I can use?

share|improve this question

5 Answers 5

up vote 2 down vote accepted

Though you asked to find the BOM, using file might even give you results when such BOM is not present. From man file:

If a file does not match any of the entries in the magic file, it is examined to see if it seems to be a text file. ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets (such as those used on Macintosh and IBM PC systems), UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC character sets can be distinguished by the different ranges and sequences of bytes that constitute printable text in each set. If a file passes any of these tests, its character set is reported.

Hence, for example:

find . -type f -exec file --mime {} \; | grep "charset=utf-16"
share|improve this answer

You can use grep:

 grep -rl $(echo -ne '^\0376\0377') *

(Tested with bash and GNU grep, might work with others.)

Explanation:

The $(echo... part generates the BOM (Hex FE FF, as octal escape sequences), this is then fed to grep as its pattern, prepended with '^' (=match start of line).

-r is recursive search, -l makes grep print the names of files it found (instead of the matching line).

This might be a bit wasteful, as grep will scan each file completely, rather than just the start. If it's mostly small text files, it will not matter. If you have loads of files with several MB, you'll have to write a perl script :-).

Alternatively, you could try file (combined with find+xargs). file will identify UTF-16 (as "UTF-16 Unicode character data"). I don't know how reliable it is, however (as it uses heuristics).

share|improve this answer
1  
"...as grep will scan each file completely..." - only the ones who do not match. -l makes grep stop at the first occurrence. With -I one also makes grep ignore binary files. –  Daniel Andersson May 22 '12 at 10:01

If you have it, you can use enca:

enca -L none * 2>/dev/null | grep  -i "Universal character"
share|improve this answer

Thanks for the help everybody. What worked best on my Mac was:

find . -type f -exec awk -F '\n' '/^\xFE\xFF|\xFF\xFE/ { print FILENAME; nextfile } { nextfile }' {} \;

It's based on sleske's solution, but takes into account that the Byte Order Mark can be reversed. It also uses awk to stop looking for the BOM after the first line, since the BOM must be at the beginning of the file. The \x escaping used to specify the BOM works with bash, I don't know if it works with other shells.

The enca tool suggested by ghostdog74 also will do the job, but it wasn't present on my Mac.

share|improve this answer

Here is the script that I use to find UTF-16 files, and subsequently convert them to UTF-8. #!/bin/sh

find ./ -type f |
while read file; do
    if [ "`head -c 2 -- "$file"`" == $'\xff\xfe' ]
    then
        echo "Problems with: $file"
        # If you want to convert to UTF-8 uncomment these lines.
        #cat "$file" | iconv -f UTF-16 -t UTF-8 > "$file.tmp"
        #mv -f "$file.tmp" "$file"
    fi
done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.