1

I would like to make sure that all of my files are correctly encoded in UTF-8 in a big project repository. Is there a tool for that or a way to do it using unix tools?

Improve this question

1 Answer 1

Reset to default
1

In general, there is no way to do this. UTF-8 has no "magic number" or marker, so you can only prove that a file is not in UTF-8 (if it contains invalid sequences), but not that it is.

You can however use a heuristic approach. What exactly works will depend on your data.

One idea:

  • Make a list of all files that are text files and contain non-ASCII characters. The second part is easy to do using perl or similar; the first will depend on what files you have. Unix file will also check for non-ASCII characters, but it's less reliable (only checks start of file).
  • If the list is small, check files manually. Otherwise, check which are valid UTF-8 (again, perl has modules for this, or use a tool like iconv or recode). The valid UTF-8 files are probably OK. The rest will have to be checked by hand (unless you know for certain how they are encoded).
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.