Each core is individually fast, but they all need access to shared resources such as caches and memory. As a result multi-core work increases the amount of work you can do, but also can cause bottlenecks.
Your mathematical calculations probably are very memory constrained and you are hitting the limit of your memory bandwidth.
That it takes 5.5 seconds on one core, but using two cores takes 6.3 seconds means you did twice as much work in only 0.8 seconds longer rather than the 5.5 seconds longer (11 seconds total) that doing the work serially would have taken. You effectively saved yourself 4.7 seconds of real world time (11 - 6.3) by doing the work side by side instead of one after the other.
Multi core does not mean you can perfectly do twice the amount of work in the same amount of time, but it does mean that you can achieve a meaningful amount more than you could have achieved with only one core.
If it now takes 10 seconds for all processes to finish when using all 4 cores then it has still finished faster than if you had done the work four times on a single core, where it should have taken 22 seconds. You are achieving four times the work in less than twice the time.
