My first suspicion for symptoms like this is a subtle RAM problem -- since OS X and Windows use RAM somewhat differently, it's plausible that it might crash consistently in one and work (almost) perfectly in the other. I've also seen flaky RAM pass "full" memory tests several times...
If you can boot OS X in single-user mode (hold command-S as it begins to boot, and it'll drop you into a very minimal command-line environment), I'd suggest running memtest on it, as I've seen it find memory problems that none of my other test utils (including the ones Apple gives service providers) would catch. Mind you, installing memtest will be a little tricky, since networking and even additional drives/partitions aren't available in single-user mode. Since it sounds like you have an external OS X boot disk, if you can mount that on another Mac, install memtest on it, and then boot the problem Mac from it, that should do it.
Another possibility is to try changing the RAM config, to remove possibly flaky RAM -- if the Mac has multiple DIMMs installed, try removing one, then the other. Try moving DIMMs between slots. If you have any spare DIMMs, try those instead of what's in the computer now.
Other things to try including booting in safe mode (hold shift as it begins to boot) -- this runs a stripped-down config with (among other things) noncritical kernel extensions disabled, so if it boots that way it may give you a some idea where the problem's coming from.