If I don't want to use Firefox, and don't want to close the process of it, and I need free memory, then how can I really hibernate the process so the memory reserved by Firefox is freed up?
For Windows, there is Free Process Freezer.
For Linux and Mac OS X, there is CrypoPID, as mentioned in yeedl's answer.
If you want to come up with a custom solution:
Look into process dumping
on Windows, eg. ProcDump, resume it with a CLI debugger from WinDBG.
Look into core dumping
on Linux, eg. core, resume it with a CLI debugger like GNU GDB.
I bet a software based solution that doesn't involve a debugger is possible, but that's for Stack Overflow.
You're looking for a very specific use case, why don't just suspend and let it swap out?
You would be spending more time on getting your processes running and suspending...
I believe you can optimize your workflow in another way that would be more seamless.