I have a corrupted disk image file (about 27 GB) in which before all \n characters \r\r was inserted. I want to remove these \r\r before all \n.

I tried with awk:

awk '{ sub("\r\r$", ""); print }' mangled.raw > image.raw

But the file seems too large: "awk: run time error: out of memory"

I also tried with sed:

sed 's/\r\r$//g' mangled.raw > image.raw

But here the output file seems incomplete: It is only 20 GB in size and the end of mangled.raw contains a lot of zero characters while the end of image.raw contains the contents of a file. Somehow sed seems to stop before the end.

Any idea how to do this right?

  • Even if you get a command to work on a file of that size, you're pretty sure to end up with a corrupted image still: in any place where there originally was a '\r\r\n', the '\r\r' will also get removed. In random binary data this sequence is expected to occur once in every 16MB, so very likeky in 27GB. – Jaap Eldering Mar 18 '14 at 11:59
  • @eldering In my file every '\n' was replaced by '\r\r\n'. This means an already existing sequence of '\r\r\n' became '\r\r\r\r\n'. A replacement of '\r\r\n' by '\n' thus restores the original state. – Mel Mar 18 '14 at 14:11
  • indeed, you are right. My mistake. – Jaap Eldering Mar 18 '14 at 14:49

eldering's comment may be correct - it depends on how the corruption happened. If it did the equivalent of s/\n/\r\r\n/ then it's reversible, but if it did s/\r*\n/\r\r\n/ then it's not.

In any case I'd use perl for something like this. Unlike sed, it was designed from the beginning to work with strings that are very long and can contain NULs and other non-text characters.

perl -pe 's/\r\r\n/\n/g' mangled.raw > image.raw

That could eat a lot of memory since it's still reading the file as a series of lines, and there could be large segments of the file with no \n that will be seen as a single "line". But if you read it by blocks you have to be careful not to miss a \r\r\n sequence that straddles a block boundary. Like this:

perl -e '
  while(<>) {
    if(/\r\z/) {
      if(length($nextblock=<>)) {
' mangled.raw > image.raw

Edit: I realized the above code would get stuck in an infinite loop if the last byte of the input was \r. It has been updated to handle that case correctly.

Edit 2: The perl one-liner contained an incorrect replacement character. It has be updated.

  • I gave your perl one-liner a try. It had the same problem as my sed one-liner: final file size was too small. However, it failed much more gracefully than sed. - It gave an out of memory error where sed pretended everything was ok. I am trying out your block implementation now. – Mel Mar 18 '14 at 16:29
  • Your block implementation worked perfectly! Many thanks. I would never have been able to write this script. – Mel Mar 18 '14 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.