Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I would like to make sure that all of my files are correctly encoded in UTF-8 in a big project repository. Is there a tool for that or a way to do it using unix tools?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

In general, there is no way to do this. UTF-8 has no "magic number" or marker, so you can only prove that a file is not in UTF-8 (if it contains invalid sequences), but not that it is.

You can however use a heuristic approach. What exactly works will depend on your data.

One idea:

  • Make a list of all files that are text files and contain non-ASCII characters. The second part is easy to do using perl or similar; the first will depend on what files you have. Unix file will also check for non-ASCII characters, but it's less reliable (only checks start of file).
  • If the list is small, check files manually. Otherwise, check which are valid UTF-8 (again, perl has modules for this, or use a tool like iconv or recode). The valid UTF-8 files are probably OK. The rest will have to be checked by hand (unless you know for certain how they are encoded).
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.