The following information is taken from the FFmpeg vpx (WebM) Encoding Guide and gives a basic overview of WebM encoding. As the libvpx encoder for FFmpeg hasn't really been documented before, any suggestions for improvement are welcome, both here and on the FFmpeg Wiki.
Constant Bit Rate
First of all, libvpx offers constant bitrate and variable bitrate encoding modes. Constant bitrate should be avoided whenever possible (unless you target a specific file size or streaming scenario), since the average quality per file size will be worse. Still, you could try setting a constant bitrate if nothing else works for you, e.g. with 1 MBit/s:
ffmpeg -i input.mp4 -c:v libvpx -minrate 1M -maxrate 1M -b:v 1M output.webm
Look at the output and increase or decrease the bit rate to your liking (or file size constraints). For example, you can use 500K or 2M, et cetera.
You have to specify -minrate, -maxrate and the bitrate -b:v in order for the encoder to use CBR. They all have to have the same value—otherwise it'll choose a different target bitrate instead and do VBR encoding, but with bad quality.
Variable Bit Rate
You need to play around with the options if you want VBR encoding, which gives you a better overall quality, since the encoder can freely choose how many bits to assign to a frame.
In libvpx, you have four parameters to control VBR encoding. All are optional, but you should set them regardless:
-qmin – the minimum quantization parameter (default 4)
-qmax – the maximum quantization parameter (default 63)
-b:v – the target bit rate setting. If not set, the encoder will choose ~1000 kBit/s as a default, but only when the -crf option is used.
-crf – the overall quality setting. If not set, the encoder will do "normal" VBR, trying to reach the target bitrate in within the qmin/qmax bounds. If set, the encoder will use CQ mode, and the target bitrate will become the maximum allowed bitrate. The CRF value is 10 by default.
Important: If neither -b:v nor -crf are set, the encoder will use a default bitrate of 256 kBit/s and your result will look bad.
Per the WebM Wiki, -qmin should be set from 0–4, and -qmax from 50–63. These Q values are quantization parameters, and lower generally means "better quality". If you set the bounds from 0 to 63, this means the encoder has free choice of how to assign the quality.
Here are different approaches on how to control the quality:
You can choose a lower -qmax value, and leave -qmin at something between 0 and 4. From a quick test it seems 20–30 are sane values here if you really want higher quality. Note that the target bit rate here is still very low and we only force it to be higher because we artificially restrict the encoder—doesn't seem like the best approach.
ffmpeg -i input.mp4 -c:v libvpx -qmin 0 -qmax 20 output.webm
A better way: You can set the -crf option and a maximum bit rate, then let libvpx do its job. To get better quality, simply choose a lower -crf value:
ffmpeg -i input.mp4 -c:v libvpx -crf 10 -b:v 2M output.webm
You can additionally specify -qmin and -qmax again, which seems like the ideal approach, because you also tell it not to drop below a certain quality (e.g., 50 here):
ffmpeg -i input.mp4 -c:v libvpx -qmin 0 -qmax 50 -crf 10 -b:v 2M output.webm
Audio
In all of the above examples, Vorbis audio should be used, by setting -c:a libvorbis. The quality can also be set to VBR, with the -q:a option.
ffmpeg -i input.mp4 … -c:a libvorbis -q:a 4 output.webm
Values range from 0 to 10, where higher means better quality. 4 should be a reasonable default and corresponds to roughly 128 kBit/s for stereo audio.
Do not use the -c:a vorbis option, as this is the FFmpeg-native encoder with very bad quality.