1

I have a batch of text files of indeterminate encoding that I need to clean of unusual characters. I've used python's chadet library to establish that they 87% conform to ISO-8859-2, but they still contain non-conforming characters which prevent them reading into R - described in this SO post. I wonder if there is a way - preferably a command line method - to batch clean these and convert to say UTF-8, with any non-confirming characters being changed to something like '~'. Very grateful for assistance.

Improve this question

1 Answer 1

Reset to default
3

Have you tried with iconv? I don't know if OSX contains this command but here's an example:

iconv -t UTF-8 myfile.txt

If that fails, try to transliterate it:

iconv -t UTF-8//TRANSLIT myfile.txt

Update (from the comments):

After a bit of experimentation iconv -t UTF-8//TRANSLIT -c infile > outfile.txt did the job. Thanks guys.

Improve this answer
6
  • Alas I get iconv: SMKA121212 copy:13:121: cannot convert
    – geotheory
    Oct 13, 2013 at 13:23
  • Edited my answer, try with //TRANSLIT as shown above
    – Savvas Radevic
    Oct 13, 2013 at 13:28
  • OS X iconv does not seem to support transliteration.
    – Daniel Beck ♦
    Oct 13, 2013 at 13:34
  • 2
    After a bit of experimentation iconv -t UTF-8//TRANSLIT -c infile > outfile.txt did the job. Thanks guys.
    – geotheory
    Oct 13, 2013 at 14:44
  • 2
    The libiconv implementation of iconv that comes with OS X does support //TRANSLIT and //IGNORE. See man iconv_open.
    – Lri
    Oct 13, 2013 at 18:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .