32

I have two big trees, which I want to compare. Some of the files in the tree differ just because one has newline at the end, and the other file lacks this newline. I want to ignore this fact. I have tried calling diff like this:

diff --ignore-all-space -r <dir1> <dir2>

And this is working. My problem is that it also ignores other differences (space-related), which may be important.

In summary: I just want to ignore the newline at EOF. Is this possible with diff?

7 Answers 7

19

You basically need to compare two files, conditionally ignoring the trailing byte. There isn't a 'diff' option to do this -- but there are a number of ways it could be done (e.g, hex diff comes to mind as well.)

To use 'diff', you basically have to modify the files that are missing the newline at the end of a file, and then compare. You could create a temporary directory with the modified files, or with a little bit of scripting it could be done in memory. (As to which is preferred depends on preference, file size, number of files...)

For example, the following will modify a file's contents (use sed -i to modify in-place, this just prints to stdout) to add a newline if one is missing (or leave the file unchanged if there already is a newline):

sed -e '$a\'  file1.txt

And just to review 'diff' syntax (returning true means they are the same, false means different):

$ diff a/file1.txt   b/file1.txt  \
      && echo '** are same' || echo '** are different'
2c2
< eof
---
> eof
\ No newline at end of file
** are different

Verify that only whitespace is different:

$ diff --ignore-all-space  a/file1.txt   b/file1.txt \
     && echo '** are same' || echo '** are different'
** are same

In bash, we can use 'sed' to manipulate the file contents as it is passed to 'diff' (original files left unchanged):

$ diff <(sed -e '$a\' a/file1.txt) <(sed -e '$a\' b/file1.txt) \
     && echo '** are same' || echo '** are different'
** are same

Now all you have to do is emulate diff -r to recursively compare directories. If comparing directories a and b, then for all files in a (e.g., a/dir1/dir2/file.txt) derive path to file in b (e.g., b/dir1/dir2/file.txt) and compare:

$ for f in $( find a -type f  )
> do
>    diff <(sed -e '$a\' $f) <(sed -e '$a\' b/${f#*/})
> done

A slightly more verbose version:

$ for f in $( find a -type f  )
> do
>   f1=$f
>   f2=b/${f#*/}
>   echo "compare: $f1 $f2"
>   diff <(sed -e '$a\' $f1) <(sed -e '$a\' $f2) \
>       && echo '** are same' || echo '** are different'
> done && echo '** all are same' || echo '** all are different'
compare: a/file1.txt b/file1.txt
** are same
compare: a/file2.txt b/file2.txt
** are same
** all are same
3
  • could you please explain what sed -e '$a\' exactly does? thx Sep 7, 2016 at 13:48
  • run sed, given the following (-e) script/expression, which matches the end of the file ($), and perform the "append" action (a\), but don't actually specify any text (nothing after the `\`) which is still going to add an EOF/newline to the end of the file (only if it's missing).
    – michael
    Sep 8, 2016 at 3:43
  • thx. I have not seen a\  yet. Sep 8, 2016 at 15:10
4

It's quite unusual that an old thread appears as the top result when searching for suppress no newline at end of file while diff on Google. What's even more surprising is that after going through all the replies, there doesn't seem to be a definitive answer to this question for over 11 years!

So, I decided to try a different approach and ran the command diff --help|grep trail. It shows

-Z, --ignore-trailing-space     ignore white space at line end

It works and I guess, this could be helpful enough when comparing batches of code that are located in multiple nested directories.

3

I solved the problem by adding a newline to each of the files and ignoring blank lines in the diff (option -B). This solutions may not be suitable for your use case but it might help others:

echo >> $FILE1 
echo >> $FILE2
diff -B $FILE1 FILE2 
2
  • I had to do this too. Yuck. I don't see the logic in diff reporting the lack of a final newline when neither file has one. Feb 8, 2020 at 14:38
  • This would be a good answer, except that diff -B is essentially broken, because (long story short) diff carries out hunk processing before line matching. If you actually have (other) blank lines in your two files, and you want the files to compare the same after these blank lines have been removed, then diff -B can give the wrong answer.
    – EML
    Oct 3, 2021 at 10:18
1

The answer is simple.
The message about the missing newline is not in the output stream of diff but in the error stream. So bend it to nirvana and you are done for good

diff -rqEeB fileA fileB 2> /dev/null
2
  • diff returns a value != 0 if it finds differences and I want to check that value. Redirecting to /dev/null does not make diff forget about that difference, so the value returned is != 0, which I do not want. I want diff to consider two files equal if the only difference is the last newline
    – blueFast
    May 21, 2015 at 7:04
  • Answer is not correct on ubuntu 20.04. Diff sends this message to standard output (stdout).
    – Dennis
    May 3, 2023 at 3:08
0

Pipe the output of diff to a grep command that drops the message you don't want to see.

1
  • not good. diff -r exists with result != 0 if I do not add --ignore-all-space. To be clear: I want diff to ignore newlines at EOF, and only at EOF. And I want it to report a result which matches this criteria. That is, if files in the tree differ only on the newline at EOF, that must not be considered a difference, and thus diff must return 0.
    – blueFast
    Feb 11, 2012 at 12:10
0

Just thought of a different approach, too, that will work for larger files (and still doesn't copy or modify the original files). You would still have to emulate the recursive directory traversal (and there are a number of ways to do that), but this example doesn't use 'sed', but rather just compares two files, excluding the last byte, using cmp, e.g.,

$ cmp  a/file1.txt  b/file1.txt  && echo '** are same' || echo '** are different'
cmp: EOF on b/file1.txt
** are different

$ du -b a/file1.txt  b/file1.txt 
13  a/file1.txt
12  b/file1.txt

$ cmp  -n 12 a/file1.txt  b/file1.txt  && echo '** are same' || echo '** are different'
** are same

Still loop over all files in the directory, and for two files a/file.txt and b/file.txt, calculate the larger file size, and subtract one, then do a binary diff (cmp) using this number of bytes (also in bash):

(( bytes = $(du -b a/file.txt  b/file.txt  | sort -nr | head -1  | cut -f1) - 1 ))
cmp -n $bytes a/file.txt b/file.txt

Looping over the files would be the same as in the other answer using sed and diff.

-1

There is a flag in diff commnad: --strip-trailing-cr that do exactly what you asked for

3
  • -1. Have you tried this? It treats /r/n like /n and has nothing to do with extra /n just before EOF. Oct 9, 2017 at 12:49
  • I've tried this, and used it to diff file with different dos/unix newline...isn't it correct?
    – dharman
    Oct 10, 2017 at 7:47
  • The question is about ignoring the newline at EOF (end of file) only. Oct 10, 2017 at 7:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .