CPUs are SISD, GPUs are SIMD.
SISD is an acronym for Single Instruction, Single Data. CPUs are good in performing sequential operations: take this, do that, move it there, take another one, add them both together, write to a device, read response, and so on. They execute mostly simple operations that take one or two values and return one value.
SIMD is Single Instruction, Multiple Data: the same operation is executed on multiple datasets simultaneously. For example take 128 values X1-X128, take 128 values Y1-Y128, multiply corresponding values in pairs and return 128 results. A SISD processor would have to execute 128 instructions (+ memory reads/writes) because it can only multiply two numbers at once. SIMD processor does this in few steps or maybe even in one if only 128 numbers fit in its register.
SISD CPUs work well for everyday computing because it's mostly sequential, but there are some tasks that require crunching large amounts of data in a similar way - for example processing graphics, video rendering, cracking passwords, mining bitcoins etc. GPUs allow for massive parallelization of computing, provided that all data must be processed in the same way.
Okay, that's pure theory. In the real world regular CPUs offer some SIMD instructions (SSE), so some multiple data stuff can be done more efficiently on a regular CPU. At the same time not all ALUs in GPUs have to work on the same thing because they are grouped into batches (see Mokubai's answer). So CPUs are not purely SISD and GPUs are not purely SIMD.
When is using GPU for computations beneficial? When your calculations are really, really massively parallelizable. You have to consider that writing input into GPU's memory takes time and reading results takes some time too. You can get the biggest performance boost when you can build a processing pipeline that does a lot of calculations before leaving GPU.