4

I want to sync two directories. The first one has CRLF and normal line endings, the second directory also has files with CRLF and normal line endings.

The problem is, when I execute this code:

  rsync -azr --exclude=images --dry-run --delete --checksum --out-format="/%f" /dir1 /dir2

It shows that a lot of files that are identical but with different encoding will be synced, and I only want to sync the different content files.

With diff, it is possible to do this:

diff --strip-trailing-cr file1 file2

But I could not find anything like this for Rsync. How can I sync only the files with different content?

  • 1
    Can you sanitize files in both directories? – Kamil Maciorowski Sep 14 '18 at 9:09
  • Welcome to Super User Sergio! An interesting question- I have edited it to make the phrasing and formatting more streamlined, feel free to roll back if you prefer :) – bertieb Sep 14 '18 at 11:06
5

There is no option to ignore line endings for rsync

As you have found out, rsync considers files with differing line endings to be different. This is inconvenient in your situation because the files are visually/semantically the same.

rsync decides what to sync in a file based on checksums on blocks. There is a good overview on this:

(T)he old version of the file is split into blocks of, e.g. 1024 or 2048 bytes, and a checksum is calculated for each block.

The new file is then searched byte for byte for blocks with checksums matching those in the old version. Here is a diagram illustrating this process:

diffing 1

Repeating these operations on the new version of the file you will iterate through the file byte for byte. During this iteration you will find two types of data in the file:

  • Blocks of data that matches blocks in the old file.
  • Sequences of bytes that is not part of a matching block.

From RSync - Detecting File Differences by Jakob Jenkov.

If you are interested, the next section is on the checksums used. However, the main point of the checksum is that it works on bytes, and your files have different bytes because of the line endings. As such, rsync is correctly detecting that they are different and so is transferring them.

Avoid transferring files with differing line endings by sanitising pre-transfer

The best say to do this is to make sure all your files have consistent line endings, or sanitise them, as Kamil suggested in the comments.

How you do this is up to you. You might decide that you will make the change when the files are generated, edited or updated. Or you could do it as a pre-transfer step.

Apply the sanitisation to only the files needed

If you do sanitise, make sure you do not apply it blindly, as Kamil further warns:

One shouldn't use any conversion tool blindly on all files. Even if the tool tries to guess if a file is text or binary, it's only heuristics. CRLF may appear inside a binary file; blocks that look like text may appear as well. Modifying a binary file by removing some bytes will most probably corrupt it.

(emphasis mine)

For example, if you have files in the two directories that you know are text files which need to be sanitised, apply the sanitisation step to that subset only.

Which tool should I use to change the endings?

A full solution is beyond the scope of this answer. There are several suggestions over on this SO QA, including dos2unix, tr, sed, awk, perl.

For example:

You can use tr to convert from DOS to Unix; however, you can only do this safely if CR appears in your file only as the first byte of a CRLF byte pair. This is usually the case. You then use:

tr -d '\015' <DOS-file >UNIX-file

However, if you're going to have to do this very often (more than once, roughly speaking), it is far more sensible to install the conversion programs (e.g. dos2unix and unix2dos, or perhaps dtou and utod) and use them.

from Jonathan Leffler's answer.

Do bear in mind the above warning when using any of those tools, however.

  • @KamilMaciorowski very true; I shied away from making any specific recommendations as it's out of scope for this question, not knowing OP's setup. I will update the answer with your warning too, however. – bertieb Sep 14 '18 at 11:55
  • Nice answer man!! – Pimp Juice IT Oct 9 '18 at 1:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.