0

I've got a rather large CSV file (~700MB) which I know to consist of lines of 27-character alpha-numeric hashes; no commas or anything fancy. Somehow, during its migration from Windows to Linux (via winSCP and then a few regular SCPs), it has converted into some kind of binary format I am unfamiliar with.

If I open the file in vi, everything appears fine, and it says [converted] at the bottom, although I know it's not a line endings issue (and dos2unix doesn't help). If I 'head' the file, it looks proper except for a "ÿþ" at the beginning of the first line. If I open up the file in nano, however, I see the "ÿþ" at the start and then "^@" before every character (even newlines and EoF).

If I try to re-save or copy the file (say via: head file.csv > short.txt), this special encoding is preserved. I copied the first ten lines out of vi (which displays it properly) into my Windows clipboard via my SSH client, then pasted it into a new text file, test.txt. This file is visually identical when opened in vi (and similar through 'head', minus the "ÿþ"), although it's roughly half of the filesize. Additionally,

file test.txt
test.txt: ASCII text
file short.txt
short.txt:

I have no idea what format this once-text file got converted to (it's notoriously hard to search the internet for symbols), but surely there must be some way to convert it back. Any ideas?

  • It might help to post short.txt somewhere online and link to it so we have a sample to work with. It sounds like some sort of text encoding change (ASCII to UCS-2? just a wild guess) but I couldn't really say for sure without seeing a bit of the data. – David Z Dec 28 '10 at 0:30
2

The "ÿþ" could be a Unicode Byte Order Mark (BOM) and the ^@ could be the first byte of each 16-bit Unicode character.

Use recode or iconv to convert it from UTF-16 to ASCII.

  • Thanks a billion! That did the trick with short.txt, although I suspect it's gonna take a little while on the full size data set. – Hammer Bro. Dec 28 '10 at 1:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.