I have two text files where the order of lines does not matter. I want to find out the lines that are different in one file to the other. Using diff in linux finds the difference but also accounts for the change in the lines of the same text.
In other words, How do I do a line insensitive difference?
If your files are not too large, you can sort them before diffing.
So something like this would work:
sort file1 > file1_sorted
sort file2 > file2_sorted
diff file1_sorted file2_sorted
Or, as you have found out in the comments, you can also do it in one line,
diff <(sort file1) <(sort file2)
Similar to the diff options you could use the comm command with the process substitutions and sort (comm requires the files it operates on be sorted). This would let you focus on just the common lines, just the lines that are in one file or the other by using -1 to supress lines only in the first file, -2 to suppress lines only in the second file or -3 to suppress lines in both, so to see all lines that are only in the first file and not the second you could do
comm -23 <(sort file1) <(sort file2)
of course, if you want that you could also consider using grep like
grep -vFxf file2 file1
which will treat the lines in file2 as patterns and only print the lines from file1 which have no matches (-v) when treating the lines from (-f) file2 as fixed strings (-F) that must match the entire line (-x).
The grep solution has the advantage of not needing to sort the files, but it only checks for lines in one of the files that are not in the other, not in both directions at once.
comm needs sorted files. (Yes, you mention it indirectly, but it's easy to overlook the way you have phrased it). Also, comm is much more efficient for this scenario than diff. The grep solution can easily run out of memory if file2 is large (exponential blowup when converting the NFA to a DFA), we had a question about this not long ago.