Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

There are many plain text files which were encoded in variant charsets.

I want to convert them all to UTF-8, but before running iconv, I need to know its original encoding. Most browsers have an Auto Detect option in encodings, however, I can't check those text files one by one because there are too many.

Only having known the original encoding, I then can convert the texts by iconv -f DETECTED_CHARSET -t utf-8.

Is there any utility to detect the encoding of plain text files? It DOES NOT have to be 100% perfect, I don't mind if there're 100 files misconverted in 1,000,000 files.

share|improve this question

7 Answers 7

up vote 12 down vote accepted

Try the chardet Python module.

Its original webpage is now dead but still available on archive.org.

It's also available on PyPi so you can install it in your python environment using

pip install chardet
share|improve this answer
    
Yes, and it's already packaged as python-chardet in Ubuntu universe repo. –  Xiè Jìléi Jun 25 '11 at 6:21
    
If it wasn't a perfect guess, chardet will still give the most correctly guess, like ./a.txt: GB2312 (confidence: 0.99). Compared to Enca which just failed and report 'Unrecognized encoding'. However, sadly enough, chardet runs very slow. –  Xiè Jìléi Jun 25 '11 at 6:48
1  
@谢继雷: Have it run overnight or something like that. Charset detection is a complicated process. You could also try the Java-based jChardet or ... the original chardet is part of Mozilla, but only C++ source is available, no command-line tool. –  grawity Jun 25 '11 at 12:13
    
Exactly what I needed. Thank you. –  Ian Hunter Apr 17 '12 at 22:36

For Linux, there is enca and for Solaris you can use auto_ef.

share|improve this answer
    
Enca seems too strict for me: enca -d -L zh ./a.txt failed with message ./a.txt: Unrecognized encoding Failure reason: No clear winner. As @grawity mentioned, chardet is more lax, however it's yet too slow. –  Xiè Jìléi Jun 25 '11 at 7:06
7  
Enca completely fails the "actually does something" test. –  Michael Wolf Mar 1 '12 at 18:59

UTFCast is worth a try. Didn't work for me (maybe because my files are terrible) but it looks good.

http://www.addictivetips.com/windows-tips/how-to-batch-convert-text-files-to-utf-8-encoding/

share|improve this answer

I would use this simple command:

encoding=$(file -bi myfile.txt)

Or if you want just the actual character set (like "utf-8"):

encoding=$(file -bi myfile.txt | sed -e 's/.*[ ]charset=//')

share|improve this answer
2  
Unfortunately, file only detects encodings with specific properties, such as UTF-8 or UTF-16. The rest -- oldish ISO8859 or their MS-DOS and Windows correspondents -- are listed as "unknown-8bit" or something similar, even for files which chardet detects with 99% confidence. –  grawity Oct 28 '11 at 19:09
2  
file showed me iso-8859-1 –  cweiske Mar 30 '12 at 7:22
    
What if the extension is lying? –  james.garriss Oct 3 at 13:24
    
@james.garriss: file extension has nothing to do with its (text) content encoding. –  MestreLion Nov 28 at 12:18

On Debian-based Linux, the uchardet package provides a command line tool. See below the package description:

 universal charset detection library - cli utility
 .
 uchardet is a C language binding of the original C++ implementation
 of the universal charset detection library by Mozilla.
 .
 uchardet is a encoding detector library, which takes a sequence of
 bytes in an unknown character encoding without any additional
 information, and attempts to determine the encoding of the text.
 .
 The original code of universalchardet is available at
 http://lxr.mozilla.org/seamonkey/source/extensions/universalchardet
 .
 Techniques used by universalchardet are described at
 http://www.mozilla.org/projects/intl/UniversalCharsetDetection.html
share|improve this answer
1  
Thanks! From the project's homepage it wasn't obvious to me that there was a CLI included. It's also available on OS X when installing uchardet via Homebrew. –  Stefan Schmidt Jul 6 '13 at 14:47
    
I was a little confused at first because a ISO 8859-1 document was falsely identified as Windows-1252 but in the printable range Windows-1252 is a superset of ISO 8859-1 so conversion with iconv works fine. –  Stefan Schmidt Jul 6 '13 at 14:56

Mozilla has a nice codebase for auto-detection in web pages:
http://lxr.mozilla.org/seamonkey/source/extensions/universalchardet/src/

Detailed description of the algorithm:
http://www-archive.mozilla.org/projects/intl/UniversalCharsetDetection.html

share|improve this answer

Getting back to chardet (python 2.?) this call might be enough:

python -c 'import chardet,sys; print chardet.detect(sys.stdin.read())' < file
{'confidence': 0.98999999999999999, 'encoding': 'utf-8'}

Though it's far from perfect....

echo "öasd" | iconv -t ISO-8859-1 | python -c 'import chardet,sys; print chardet.detect(sys.stdin.read())'
{'confidence': 0.5, 'encoding': 'windows-1252'}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.