0

I have these 3 files

21/08/2014  07:15 PM               122 Tw2AWK.csv
21/08/2014  07:15 PM               125 Tw2Notepad.csv
21/08/2014  07:15 PM               119 Tw2REPL.csv

C:\myfiles>file Tw2AWK.csv TwREPL.csv Tw2Notepad.csv

Tw2AWK.csv;     UTF-8 Unicode text, with CRLF line terminators
Tw2REPL.csv;     UTF-8 Unicode text
Tw2Notepad.csv; UTF-8 Unicode (with BOM) text, with CRLF line terminators

HEX of these files is as follows

C:\myfiles>xxd -p Tw2REPL.csv

0a222344656c686947616e675261706520776173206120736d616c6c2069
6e636964656e7420746f2023536d616c6c5261706973744a6169746c6579
20646e61696e6469612e636f6d2f696e6469612f7265706f72742d69e280
a6207069632e747769747465722e636f6d2f6762565070776637744f22

C:\myfiles>xxd -p Tw2AWK.csv

0d0a222344656c686947616e675261706520776173206120736d616c6c20
696e636964656e7420746f2023536d616c6c5261706973744a6169746c65
7920646e61696e6469612e636f6d2f696e6469612f7265706f72742d69e2
80a6207069632e747769747465722e636f6d2f6762565070776637744f22
0d0a

C:\myfiles>xxd -p Tw2Notepad.csv

efbbbf0d0a222344656c686947616e675261706520776173206120736d61
6c6c20696e636964656e7420746f2023536d616c6c5261706973744a6169
746c657920646e61696e6469612e636f6d2f696e6469612f7265706f7274
2d69e280a6207069632e747769747465722e636f6d2f6762565070776637
744f220d0a

I want Tw2REPL.csv to look like Tw2Notepad.csv How can I do it?

NOTE: I have do this all via command line . I can use any 3rd party standalone exe's though. I am on Windows XP

Please help, its very important for me

0

1 Answer 1

2

First, add the bytes ef bb bf (UTF-8 encoded U+FEFF) to the beginning of the file. It's the byte-order mark, which was invented for UTF-16 and doesn't make much sense for UTF-8, though some programs rely on it.

$ (echo ef bb bf; xxd -p file.csv) | xxd -r -p > file.csv.new

C:\> (echo ef bb bf & xxd -p file.csv) | xxd -r -p > file.csv.new

Then, convert Unix line-endings (LF 0a) to Windows format (CR LF 0d 0a).

    $ unix2dos file.csv
or: $ todos file.csv
or: $ sed "s/$/\r/" file.csv > file.csv.new

Since you have xxd, I assume you also have Vim. You can do this in Vim by opening the file, then using:

:set bomb ff=dos
:w
7
  • Sorry sir for not clarifying, but i have to do all of this via commandline (i.e batch commands). I can use any 3rd party exe's like sed, tr, xxd, though. Thanks (P.s. i not not a (very) technical user) Aug 22, 2014 at 8:01
  • @SuperUserMan: See edits. Aug 22, 2014 at 8:04
  • dang, i forgot to tell, i am on windows xp not linux :( Aug 22, 2014 at 8:33
  • @SuperUserMan: The first command remains almost the same, just change the ; to & if you use cmd.exe rather than sh/bash. The other tools (todos or sed) can be installed the same way you already installed the file or xxd commands. Aug 22, 2014 at 9:33
  • @grawity curious, I notice that xxd -r -p gives a different result to xxd -rp. pastebin.com/raw.php?i=iG5C0Jsw -r -p seems to not be funny, though you used -rp
    – barlop
    Aug 22, 2014 at 10:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .