What's the problem with Handbrake?
When you're using Handbrake to convert from one container format to
the other (i.e. MKV to MP4 in your case), Handbrake will re-encode the video. See also the respective feature request that would have enabled video passthrough:
Sorry, adding video passthrough is not planned. HandBrake is designed to be a video transcoder. It wasn't designed to allow passthrough.
So, anytime you're changing containers with Handbrake, your video is going to get re-encoded, which means it a) takes time and b) may introduce quality loss.
Do I have to re-encode? Couldn't I just swap the container?
Since passthrough is not possible, ask yourself: Do I need to re-encode? If you only want to change the container from MKV to MP4, you usually don't need to encode anything, you just change the "wrapping" around the video. This doesn't lose quality, and it'll be a much faster process.
You can swap containers easily with FFmpeg – you just have to tell it to copy the video and audio streams:
ffmpeg -i input.mkv -c copy -map 0 output.mp4
There are also tools like MP4Box which can also create MP4 containers — the same exists for MKV with MKVtoolnix.
However, there's a big caveat: this only works if the audio and video codecs are supported in the target (MP4) container, which is the case for H.264/H.265 and AAC, for example, but not for many others. Also, subtitle format support for MP4 is different from MKV, and actually quite restricted, so this command may fail.
If this command does not work, and if your input uses the wrong codecs for the output container, you will probably have to re-encode. In this case, the codecs will be adapted to the output container.
To understand why this is necessary, it's important to learn the difference between video codecs and containers. This will help you understand why changing containers works and why the containers MP4 and MKV have little to do with video codecs, actually.
Why is re-encoding bad, anyway?
You can (usually) not retain full quality when encoding a video that was already encoded. This is because the original has already been compressed by throwing away information, and by doing it again you're introducing generation loss.
Often, you want to re-encode video when for example its dimensions change, or you need a specific bit rate to squeeze your video stream into, or your original video uses a codec that you can't play for whatever reason.
So, if you load your MKV video into Handbrake, and re-encode it with x264, the H.264 encoder Handbrake uses, store it in an MP4 container, you are going to lose quality no matter what, unless you set the bitrate or quality factor so high that you won't (really) see the difference. But then, the file size will be bigger as well.
In the ideal case, you would convert the video to an uncompressed video, which won't lose you any quality, but give you files of a dozen Gigabytes in size, even for a few minutes of video material.
Okay, but I really have to re-encode!
If you really have to re-encode, make sure not to set an average bitrate, but choose a Constant Rate Factor, which is something like "constant quality". Just like "variable bit rate" for MP3: It will make sure to spend the bits on the video parts that need them and make the overall quality better — at the same file size.
Sane CRF values are from 19 to 24, where lower means "better". So, you could try with a Rate Factor of 19. Also, make sure to set the "High" profile, which enables the encoder to use all bells and whistles and optimize the quality for a given bit rate.