First of all, ask yourself: Do you need to re-encode? If you only want to change the container from MKV to MP4, you don't need to encode anything, you just change the "wrapping" around the video. This doesn't lose quality.
You can swap containers easily with FFmpeg – you just have to tell it to copy the video and audio bitstreams:
ffmpeg -i input.mkv -c:v copy -c:a copy output.mp4
There are also tools like MP4Box which can create MP4 media — the same exists for MKV with MKVtoolnix.
Finally, learn the difference between video codecs and containers. This will help you understand why changing containers works and why MP4 and MKV have little to do with video, actually. If you want to know more about FFmpeg – I wrote a blog entry on the Super User blog about it.
However, you can not retain full quality when encoding a video that was already encoded. This is because the original has already been compressed by throwing away information, and by doing it again you're introducing generation loss. Often, you want to re-encode video when for example its size changes, or you need a specific bit rate to squeeze your video stream into, or your original video uses a codec that you can't play for whatever reason.
So, if you load your MKV video into Handbrake, and re-encode it with x264, the h.264 encoder Handbrake uses, store it in an MP4 container, you are going to lose quality no matter what, unless you set the bitrate or quality factor so high that you won't (really) see the difference. But then, the file size will be bigger as well. In the ideal case, you would convert the video to an uncompressed video, which won't lose you any quality, but give you files of a dozen Gigabytes in size, even for a few minutes of video material.
If you really have to re-encode, make sure not to set an average bitrate, but choose a Constant Rate Factor, which is something like "constant quality". Just like "variable bit rate" for MP3: It will make sure to spend the bits on the video parts that need them and make the overall quality better — at the same file size.
Sane CRF values are from 19 to 24, where lower means "better". So, you could try with a Rate Factor of 19. Also, make sure to set the "High" profile, which enables the encoder to use all bells and whistles and optimize the quality for a given bit rate.