Yes, this happened lots with my MacBook; the fans seem to have an unusually high temperature threshold until they kick in. It may be because since the case is solid aluminium, it conducts the heat away from the internal components, so is in itself much hotter than the actual inside components, leading to the fans not kicking in even though the body feels hot.
I downloaded Fan Control which allows you to change the fan speed, and has been working great and keeping it cool, although sometimes it can be a little too sensitive - it depends how you set it up.
Edit: the use of the unibody to transfer heat from the internals to the air outside is certainly true for the bottom of the MacBook, probably the top as well, so this again, is probably why it seems hotter than the actual components are:
The bottom of the MacBook case functions as a cooling surface that transfers heat from inside the computer to the cooler air outside.