You might consider just taking notes on paper, then scanning them. Probably easier and faster than any software solution.
I don't think there's any software for actually entering math formulas as formulas (like in LaTeX or some formula editor); I think the only feasible option is to capture pen input as graphics. Edit: Actually, at least the "Math Input Panel" in Windows 7 apparently will do just that (handwriting recognition for formulas). Haven't tried it, but looks really nifty.
That said, you might want to check out Digital paper. It's basically a special pen & paper. You write normally, just like regular paper, but the pen digitizes "on-the-fly". You still only get graphics of your pages as output, but you save the hassle of scanning, plus it will really only digitize what you write, so no messy artifacts from fingerprints, background pattern of paper etc. which you would get from scanning notes.
An acquaintance uses one of these, and is quite satisfied.