0

I have these 3 files

21/08/2014  07:15 PM               122 Tw2AWK.csv
21/08/2014  07:15 PM               125 Tw2Notepad.csv
21/08/2014  07:15 PM               119 Tw2REPL.csv

C:\myfiles>file Tw2AWK.csv TwREPL.csv Tw2Notepad.csv

Tw2AWK.csv;     UTF-8 Unicode text, with CRLF line terminators
Tw2REPL.csv;     UTF-8 Unicode text
Tw2Notepad.csv; UTF-8 Unicode (with BOM) text, with CRLF line terminators

HEX of these files is as follows

C:\myfiles>xxd -p Tw2REPL.csv

0a222344656c686947616e675261706520776173206120736d616c6c2069
6e636964656e7420746f2023536d616c6c5261706973744a6169746c6579
20646e61696e6469612e636f6d2f696e6469612f7265706f72742d69e280
a6207069632e747769747465722e636f6d2f6762565070776637744f22

C:\myfiles>xxd -p Tw2AWK.csv

0d0a222344656c686947616e675261706520776173206120736d616c6c20
696e636964656e7420746f2023536d616c6c5261706973744a6169746c65
7920646e61696e6469612e636f6d2f696e6469612f7265706f72742d69e2
80a6207069632e747769747465722e636f6d2f6762565070776637744f22
0d0a

C:\myfiles>xxd -p Tw2Notepad.csv

efbbbf0d0a222344656c686947616e675261706520776173206120736d61
6c6c20696e636964656e7420746f2023536d616c6c5261706973744a6169
746c657920646e61696e6469612e636f6d2f696e6469612f7265706f7274
2d69e280a6207069632e747769747465722e636f6d2f6762565070776637
744f220d0a

I want Tw2REPL.csv to look like Tw2Notepad.csv How can I do it?

NOTE: I have do this all via command line . I can use any 3rd party standalone exe's though. I am on Windows XP

Please help, its very important for me

2

First, add the bytes ef bb bf (UTF-8 encoded U+FEFF) to the beginning of the file. It's the byte-order mark, which was invented for UTF-16 and doesn't make much sense for UTF-8, though some programs rely on it.

$ (echo ef bb bf; xxd -p file.csv) | xxd -r -p > file.csv.new

C:\> (echo ef bb bf & xxd -p file.csv) | xxd -r -p > file.csv.new

Then, convert Unix line-endings (LF 0a) to Windows format (CR LF 0d 0a).

    $ unix2dos file.csv
or: $ todos file.csv
or: $ sed "s/$/\r/" file.csv > file.csv.new

Since you have xxd, I assume you also have Vim. You can do this in Vim by opening the file, then using:

:set bomb ff=dos
:w
  • Sorry sir for not clarifying, but i have to do all of this via commandline (i.e batch commands). I can use any 3rd party exe's like sed, tr, xxd, though. Thanks (P.s. i not not a (very) technical user) – SuperUserMan Aug 22 '14 at 8:01
  • @SuperUserMan: See edits. – grawity Aug 22 '14 at 8:04
  • dang, i forgot to tell, i am on windows xp not linux :( – SuperUserMan Aug 22 '14 at 8:33
  • @SuperUserMan: The first command remains almost the same, just change the ; to & if you use cmd.exe rather than sh/bash. The other tools (todos or sed) can be installed the same way you already installed the file or xxd commands. – grawity Aug 22 '14 at 9:33
  • @grawity curious, I notice that xxd -r -p gives a different result to xxd -rp. pastebin.com/raw.php?i=iG5C0Jsw -r -p seems to not be funny, though you used -rp – barlop Aug 22 '14 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.