up vote 3 down vote favorite
3
share [g+] share [fb]

Possible Duplicate:
Batch-convert files for encoding or line ending

I have a bunch of text files that I'd like to convert from any given charset to UTF-8 encoding.

Are there any command line tools or Perl (or language of your choice) one liners I can use to do this en masse?

link|improve this question
1  
Hilariously enough. This was asked first. – jason May 28 '10 at 1:00
feedback

closed as exact duplicate by quack quixote, Diago Feb 25 '10 at 22:00

This question covers exactly the same ground as earlier questions on this topic; its answers may be merged with another identical question. See the FAQ.

1 Answer

up vote 9 down vote accepted

iconv does convert between many character encodings. So adding a little bash magic and we can write

for i in *.txt; do iconv -f ascii -t utf-8 "$i" -o "${i/.txt}.utf8.txt"; done

This will run iconv -f ascii -t utf-8 to every file ending in .txt, sending the recoded file to a file with the same name but ending in .utf8.txt instead of .txt.

It's not as if this would actually do anything to your files (because ascii is a subset of utf-8), but to answer your question about how to convert between encodings.

link|improve this answer
You should quote the var $i, in order to handle filenames with spaces. – Richard Hoskins Aug 1 '09 at 1:47
It will do things, it'll add a BOM for one... – jason Aug 1 '09 at 1:58
Are you sure iconv will add a BOM? I was under the impression that it wouldn't with UTF-8. – Richard Hoskins Aug 1 '09 at 2:08
1  
I just tested this with iconv (GNU libiconv 1.11), and it did not add a BOM. It is my understanding that iconv will only add a BOM if one is present in the input, which it would not be in ASCII. BOM are problematic, and not necessary with UTF-8. – Richard Hoskins Aug 1 '09 at 2:31
Thanks for the correction. – jason Aug 1 '09 at 2:59
show 2 more comments
feedback

Not the answer you're looking for? Browse other questions tagged or ask your own question.