The project https://reproducible-builds.org/ is all about this, and is trying hard to make the answer to your question "no, they will not differ" in as many places as possible. NixOS and Debian are now over 90% in reproducibility for their packages.
If you compile a binary, and I compile a binary, and they're bit-for-bit identical, then I can be reassured that the source code and the tools are what determine the output, and that you didn't sneak in some trojan code along the way.
If we combine reproducibility with bootstrappability from human-readable source, as http://bootstrappable.org/ is working on doing, we get a system determined from the ground up by human-readable source, and only then are we at a point where we can trust that we know what the system is doing.