Does recompiling a program produce a bit-for-bit identical binary?
For all compilers? No. The C# compiler, at least, is not allowed to.
Eric Lippert has a very thorough breakdown on why the output of the compiler is not deterministic.
[T]he C# compiler by design never produces the same binary twice. The C# compiler embeds a freshly generated GUID in every assembly, every time you run it, thereby ensuring that no two assemblies are ever bit-for-bit identical. To quote from the CLI specification:
The Mvid column shall index a unique GUID [...] that identifies this instance of the module. [...] The Mvid should be newly generated for every module [...] While the [runtime] itself makes no use of the Mvid, other tools (such as debuggers [...]) rely on the fact that the Mvid almost always differs from one module to another.
Although it's specific to a version of the C# compiler, many points in the article can be applied to any compiler.
First off, we are assuming that we always get the same list of files every time, in the same order. But that's in some cases up to the operating system. When you say "csc *.cs", the order in which the operating system proffers up the list of matching files is an implementation detail of the operating system; the compiler does not sort that list into a canonical order.