I have a .csv with special characters that are wrecking an import I need to do.

In VIM, I discovered away to remove the special characters with :


This removes ^M from some of my broken lines.

But I'd like to either write this into a bash script, or into my ruby script. So I was curious if there's a way I can perform this special character search and replace in either Bash or Ruby.

Searching for ^M does not work. And also you wouldn't see this character in any text editor except for VIM.

  • The simplest way to remove ^M is to use dos2unix. You will need to use an extra work file in your script. – AFH May 27 '15 at 15:06
  • Ah, would you know what syntax to use? @AFH – Trip May 27 '15 at 15:08
  • dos2unix WinFile.txt LinFile.txt. Alternatively, use sed -i 's/\r$//' WinFile.txt if you want to avoid a work file. – AFH May 27 '15 at 15:14
  • @AFH Ah yes, the sed command is definitely what I'm looking for. But using the -i option returns : sed: 1: "/Users/mycomputer/Des ...": invalid command code e Which i heard the remedy is just prepending a -e. So this passes sed -i -e 's/\r$//' ~/Desktop/my_file.csv , but the special character isn't actually removed. So no change actually occurred. – Trip May 27 '15 at 17:24
  • My command above works on Ubuntu. Other distributions may have different defaults for sed - Ubuntu defaults to Basic Regular Expressions, which include handling of \r and related escape sequences. Just a passing thought: your file isn't read-only is it? – AFH May 27 '15 at 17:50

This works for me (Fedora 20, GNU sed 4.2.2):

sed -e 's/\r$//' <file-in> > <file-out>

(Replace <file-in> and <file-out> with your file names.) I use file to check the line endings.

  • This works I think, but it just outputs the text in my terminal screen and doesn't write it to the <file-out>, perhaps I'm missing a pipe and some further instruction to do that? – Trip Jul 29 '15 at 10:24
  • 1
    Note the extra >; this is the indirection. You can also try sed -e 's/\r$//' yourfile | more or in-place substitution sed -i -e 's/\r$//' yourfile. – Atafar Jul 31 '15 at 14:14
  • I know I marked this correct already.. but I just tested this out, and it doesn't seem to move the ^M from the file.. – Trip Aug 11 '15 at 14:35
  • If you also have '^M' in other places than at the end of the line, use 'sed -e 's/^M//g' ...' – Atafar Aug 11 '15 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.