edit: Now that I got answers, I marked one by @KamilMaciorowski that fits better to the title as an answer, but this answer by @oliv actually were better suited to my actual need to my primary purpose. (To process csv file with breaks consistently on awk.)
So in the case if you were looking for awking in the similar circumstance, I recommend checking that first!
Please help me preparing a few thousands of csv file ready for
awk to process! Some of the field has line breaks inside the field and that's causing
awk to process them as a multiple records.
However those problematic line breaks only happens where ^M is inserted, so I just need to remove ^M and line-break altogether from all of them.
^Ms are indeed line break character, not literal caret & letter M string. This file is generated for .net to parse and process, but I haven't worked on developing apps on neither file producing/reading sides, so I don't really know how it's successfully parsed. It's exclusively used for fields in certain columns with multiple-lined strings (comments).
So how do you make this (csv with 1 header and 2 records. Some field has line breaks in it preceeded by ^M):
"header_1", "header_2", "header_3" "1-1", "1-2", "1-3" "2-1", "2-2_a^M 2-2_b^M 2-2_c", "2-3"
like this? (csv with 1 header and 2 records without line breaks within each of them.):
"header_1", "header_2", "header_3" "1-1", "1-2", "1-3" "2-1", "2-2_a2-2_b2-2_c", "2-3"
I tried removing them with
sed but I heard there's no way to process, and I didn't quite get the reason why.
for file in *.csv; do sed -e "s/^M//" $file > sedded/$file; done
Anyhow, I get this:
"header_1", "header_2", "header_3" "1-1", "1-2", "1-3" "2-1", "2-2_a 2-2_b 2-2_c", "2-3"
I tried to go for something like
"s/^M\n/", and it doesn't work as I suspected. Should I use completely different tool like
vim? As long as it works for thousands of files at once (each containing ~500 lines, and I don't really care the time it takes to process) I'm fine with any sort of resolution. Just thought
sed was the way. (I'm ok to use DOS command/powershell if it's easier or more straight forward!)