This started as a comment, but it became too long, so I post it as an answer.
You say you don't want Python, but Python would be quick and complete, and to compare Python and MATLAB in "full-blown"-ity is quite harsh. I believe it would also fare well speed wise compared the the C# and Perl suggestions among the comments and answers.
I use the one-liner:
python -ic "from __future__ import division; from math import *;"
which I've bound to a keyboard shortcut which opens a terminal with this command, which gives me a lightning-fast competent calculator with exp(), sqrt(), sin(), log(), pi, e, etc.
If you just want a "calc 5+7" variant, the *nix variete would be:
python -c "from __future__ import division; from math import *; print $*"
and on Windows you probably just need to replace $* with %*.
from __future__ import division makes division use floating point instead of integer division as default, which is expected of a calculator. This is not needed if one uses Python 3 (which is what the __future__ part means).
from math import * imports all math functions into the main namespace, so you can do sin(2*pi) instead of math.sin(2*pi).
As an encore: to have a script that can be started in either interactive or direct mode:
#!/bin/sh
if [ ${#} = 0 ]; then
python -ic "from __future__ import division; from math import *;"
else
python -c "from __future__ import division; from math import *; print ${*};"
fi
(most probably trivially converted to Windows).