GPU and CPU serve different purposes, and different purposes usually mean different solutions.
GPUs are massively parallelized, but they are used for completely different operations than CPUs. Those operations can be processed in parallel with good results. But not every algorithm can be parallelized. Parallel processing brings a lot of new problems that have to be solved in some way. Sometimes the overhead from those solutions is so big that it exceeds the performance gain. And some algorithms just can't be parallelized.
Multithreading isn't an ultimate solution for gaining better performance. It works in situations where multiple parallel operations have to be performed, like on database servers etc. But at home you couldn't make use of massively parallelized system. Your everyday tasks work well on few cores, a powerful single-core CPU would probably handle it too.
Now let's say you have replaced your 4-core CPU with 256-core one. More cores come at a price: each core requires additional power and more sophisticated cache mechanisms have to be implemented, thus increasing CPU's complexity. You have 64 times more cores, so each of those will be 64 times slower. Instead of 2.4 GHz per core, now you have 37.5 MHz per core.
What happens? Your system is consuming way more power and giving much more heat. If you try to use Windows 8, it's crawling. Booting takes forever, despite hybrid boot being turned on. You try Windows 98 and it's still too slow to use it. Most of your cores aren't used at all. Few are working at 100%, but it's still just 37.5 MHz (versus 2400 MHz on your previous CPU).
Of course that's an extreme example, but it's a real issue. Actually, there's some interesting discussion on this topic going on currently: Samsung has presented Exynos Octa, an eight-core mobile CPU family based on big.LITTLE architecture, and here's the Qualcomm's response.