... if you are using a subset of the original list of modules? I tried this recently and the kernel couldn't boot, and then when I did a full rebuild it apparently worked. But since I only tried this once I can't be absolutely sure that there wasn't some other thing I changed between the first and second rebuilds. I could try it again, I suppose, but it'll take time :P
There was some work to make modules (somewhat) version independent, in that each function essentially recorded its prototype and the module refused to load if there was any mismatch. I haven't followed this too closely, so I don't know if that was ripped out.
Short answer: Your safest bet is to recompile, the kernel hackers are adamant at not having a stable API (let alone ABI) inside the kernel. Even a configuration change could change data structure layout changes.