49

Possible Duplicate:
Batch-convert files for encoding or line ending

I have a bunch of text files that I'd like to convert from any given charset to UTF-8 encoding.

Are there any command line tools or Perl (or language of your choice) one liners I can use to do this en masse?

Improve this question
0

1 Answer 1

Reset to default
59

iconv does convert between many character encodings. So adding a little bash magic and we can write

for file in *.txt; do
    iconv -f ascii -t utf-8 "$file" -o "${file%.txt}.utf8.txt"
done

This will run iconv -f ascii -t utf-8 to every file ending in .txt, sending the recoded file to a file with the same name but ending in .utf8.txt instead of .txt.

It's not as if this would actually do anything to your files (because ASCII is a subset of UTF-8), but to answer your question about how to convert between encodings.

Improve this answer
10
  • 2
    You should quote the var $i, in order to handle filenames with spaces.
    – Richard Hoskins
    Aug 1, 2009 at 1:47
  • It will do things, it'll add a BOM for one...
    – jason
    Aug 1, 2009 at 1:58
  • Are you sure iconv will add a BOM? I was under the impression that it wouldn't with UTF-8.
    – Richard Hoskins
    Aug 1, 2009 at 2:08
  • 6
    I just tested this with iconv (GNU libiconv 1.11), and it did not add a BOM. It is my understanding that iconv will only add a BOM if one is present in the input, which it would not be in ASCII. BOM are problematic, and not necessary with UTF-8.
    – Richard Hoskins
    Aug 1, 2009 at 2:31
  • 2
    if your version of iconv does not support the -o parameter you can directly replace it with >> to use the shell redirection.
    – rob
    Oct 9, 2015 at 8:45

Not the answer you're looking for? Browse other questions tagged .