This is a guess: It might be that the time resolution is not large enough and the files are different while their timestamps fall into the same second and have the same length, so rsync can't distinguish them and skips them, thinking they are the same. The option
--ignore-times might help in this case, as that will force rsync to check the file content, not just file size and timestamp.
Another alternative to
--checksum, which will force rsync to checksum each file and should guarantee you that the directories are exactly equal afterwards, it will however slow things down a bit, especially on large file hierarchies.
Another more mundane issue could simply be failing hardware, RAM, harddrive or whatever that leads to files being copied incorrectly, I have seen that happen quite a few times. To check for that use a memory checker like
memtest86+ and do repeated copies of the same directory structure to a different target directory and compare the results, if the results are different each time, something is fishy.