up vote 9 down vote favorite
3
share [g+] share [fb]

There are many plain text files which were encoded in variant charsets.

I want to convert them all to UTF-8, but before running iconv, I need to know its original encoding. Most browsers have an Auto Detect option in encodings, however, I can't check those text files one by one because there are too many.

Only having known the original encoding, I then can convert the texts by iconv -f DETECTED_CHARSET -t utf-8.

Is there any utility to detect the encoding of plain text files? It DOES NOT have to be 100% perfect, I don't mind if there're 100 files misconverted in 1,000,000 files.

link|improve this question
92% accept rate
feedback

4 Answers

up vote 6 down vote accepted

Try the chardet Python module.

(2012-04-18: Website appears to be dead. It's still on Github and archive.org, though.)

link|improve this answer
Yes, and it's already packaged as python-chardet in Ubuntu universe repo. – Xiè Jìléi Jun 25 '11 at 6:21
If it wasn't a perfect guess, chardet will still give the most correctly guess, like ./a.txt: GB2312 (confidence: 0.99). Compared to Enca which just failed and report 'Unrecognized encoding'. However, sadly enough, chardet runs very slow. – Xiè Jìléi Jun 25 '11 at 6:48
@谢继雷: Have it run overnight or something like that. Charset detection is a complicated process. You could also try the Java-based jChardet or ... the original chardet is part of Mozilla, but only C++ source is available, no command-line tool. – grawity Jun 25 '11 at 12:13
Exactly what I needed. Thank you. – beanland Apr 17 at 22:36
feedback
up vote 5 down vote

For Linux, there is enca and for Solaris you can use auto_ef.

link|improve this answer
Enca seems too strict for me: enca -d -L zh ./a.txt failed with message ./a.txt: Unrecognized encoding Failure reason: No clear winner. As @grawity mentioned, chardet is more lax, however it's yet too slow. – Xiè Jìléi Jun 25 '11 at 7:06
2  
Enca completely fails the "actually does something" test. – Michael Wolf Mar 1 at 18:59
feedback
up vote 4 down vote

I would use this simple command:

encoding=$(file -bi myfile.txt)

Or if you want just the actual character set (like "utf-8"):

encoding=$(file -bi myfile.txt | sed -e 's/.*[ ]charset=//')

link|improve this answer
Unfortunately, file only detects encodings with specific properties, such as UTF-8 or UTF-16. The rest -- oldish ISO8859 or their MS-DOS and Windows correspondents -- are listed as "unknown-8bit" or something similar, even for files which chardet detects with 99% confidence. – grawity Oct 28 '11 at 19:09
1  
file showed me iso-8859-1 – cweiske Mar 30 at 7:22
feedback
up vote 0 down vote

UTFCast is worth a try. Didn't work for me (maybe because my files are terrible) but it looks good.

http://www.addictivetips.com/windows-tips/how-to-batch-convert-text-files-to-utf-8-encoding/

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.