I have these 3 files

21/08/2014  07:15 PM               122 Tw2AWK.csv
21/08/2014  07:15 PM               125 Tw2Notepad.csv
21/08/2014  07:15 PM               119 Tw2REPL.csv

C:\myfiles>file Tw2AWK.csv TwREPL.csv Tw2Notepad.csv

Tw2AWK.csv;     UTF-8 Unicode text, with CRLF line terminators
Tw2REPL.csv;     UTF-8 Unicode text
Tw2Notepad.csv; UTF-8 Unicode (with BOM) text, with CRLF line terminators

HEX of these files is as follows

C:\myfiles>xxd -p Tw2REPL.csv


C:\myfiles>xxd -p Tw2AWK.csv


C:\myfiles>xxd -p Tw2Notepad.csv


I want Tw2REPL.csv to look like Tw2Notepad.csv How can I do it?

NOTE: I have do this all via command line . I can use any 3rd party standalone exe's though. I am on Windows XP

Please help, its very important for me


First, add the bytes ef bb bf (UTF-8 encoded U+FEFF) to the beginning of the file. It's the byte-order mark, which was invented for UTF-16 and doesn't make much sense for UTF-8, though some programs rely on it.

$ (echo ef bb bf; xxd -p file.csv) | xxd -r -p > file.csv.new

C:\> (echo ef bb bf & xxd -p file.csv) | xxd -r -p > file.csv.new

Then, convert Unix line-endings (LF 0a) to Windows format (CR LF 0d 0a).

    $ unix2dos file.csv
or: $ todos file.csv
or: $ sed "s/$/\r/" file.csv > file.csv.new

Since you have xxd, I assume you also have Vim. You can do this in Vim by opening the file, then using:

:set bomb ff=dos
| improve this answer | |
  • Sorry sir for not clarifying, but i have to do all of this via commandline (i.e batch commands). I can use any 3rd party exe's like sed, tr, xxd, though. Thanks (P.s. i not not a (very) technical user) – SuperUserMan Aug 22 '14 at 8:01
  • @SuperUserMan: See edits. – user1686 Aug 22 '14 at 8:04
  • dang, i forgot to tell, i am on windows xp not linux :( – SuperUserMan Aug 22 '14 at 8:33
  • @SuperUserMan: The first command remains almost the same, just change the ; to & if you use cmd.exe rather than sh/bash. The other tools (todos or sed) can be installed the same way you already installed the file or xxd commands. – user1686 Aug 22 '14 at 9:33
  • @grawity curious, I notice that xxd -r -p gives a different result to xxd -rp. pastebin.com/raw.php?i=iG5C0Jsw -r -p seems to not be funny, though you used -rp – barlop Aug 22 '14 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.