Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
  1. I was wondering if some text files store their encoding method along their text content for later decoding?
  2. Or is it the text viewer's job to guess the encoding method for a given text file, and the guessing may not always be correct? If yes, how does a text viewer guess that?
share|improve this question
    
If it's a plaintext file, then it doesn't store anything about the encoding. I can't say for rich text, though. –  Wuffers Jul 9 '11 at 14:49
    
Yes, I am talking about plaintext. –  Tim Jul 9 '11 at 14:50

2 Answers 2

up vote 7 down vote accepted

I was wondering if some text files store their encoding method along their text content for later decoding?

Mark Szymanski's answer is correct - there is no explicit encoding information in a plain text file - that's the definition of "plain text file", the "plain" refers to the fact that there is no meta-data in the file.

However, some applications will place a byte-order mark (BOM) in text files encoded as UTF-16 or UTF-32/UCS-4. The BOM is not really meant to indicate the encoding (it indicates byte order, as the name says), but many applications will use the presence of the BOM to recognize UTF-16/UTF-32, so it serves as an encoding indicator.

Or is it the text viewer's job to guess the encoding method for a given text file, and the guessing may not always be correct? If yes, how does a text viewer guess that?

Yes, the text viewer can only guess. It usually uses some heuristics:

  • In some encodings (notably in UTF-8) not all byte sequences are valid. So an application can just try to decode the file as UTF-8. If it succeeds, the file is probably UTF-8; if it fails by finding an invalid byte sequence, it is not. This is how e.g. vim works by default: It will first try to use UTF-8 when reading a file; if that fails, it falls back to ISO-8859-1.
  • In most older 8-bit encodings, any byte sequence is valid. In that case, you can sometimes guess encoding by looking at the byte histogram (frequency of different bytes/byte sequences). Internet Explorer used to do this to "guess" the encoding of a page. However, this is very error-prone, so few programs do this.

In most cases, a program must be explicitly told what the encoding of a text file is, otherwise it will not be able to read it correctly.

share|improve this answer
    
So how does file -bi work if BOM is not used? –  Old Geezer Nov 17 at 1:57
    
@OldGeezer: file has various heuristics for determining file type and encoding. Mostly, it looks for certains strings or byte sequences in the file. If you want more specific information, you'll probably have to read the source. Or just ask a separate question :-). –  sleske Nov 17 at 16:15
    
@OldGeezer: And BTW, file cannot do reliably detect most text encodings (because that is very difficult). The man page has some information on character set detection - file mostly only recognizes ASCII, UTF-8/16, EBCDIC, and ISO-8859-x. For example, a file encoded in KOI8-R is reported as "ISO-8859-1". –  sleske Nov 17 at 16:23

Plain text files do not store any information about their encoding. A viewer determines it based on the character encoding you have set for it. It can not determine it by itself, since it's all the same to the computer.

share|improve this answer
    
So text viewers can not distinguish between encoding methods for text files. If a text viewer is given an object/executable file, will it be able to tell that it is not a text file? –  Tim Jul 9 '11 at 15:09
    
Nope, it can't. It will try to open it like a text file. And of course will display a bunch of garbled stuff. The only way you can get it to distinguish between encodings is if you manually change the encoding. –  Wuffers Jul 9 '11 at 15:10
    
@Tim: Most text viewers use a heuristic to check whether something is a text file. If the file has many non-printable characters, many viewers and editors will warn (e.g. less and grep on Unix/Linux do this). –  sleske Nov 2 '13 at 10:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.