For the 1st question:
Let me assume that the video saved as FLV was okay to play and there was nothing wrong on the FLV file. If so, possibly the AVI file you saved was not compressed at all and contained a bitmap stream which must be really heavy. See the "export", "compression" or "codec" option (or anything like that) on your tool to export the movie and there should be an option to change the codec for AVI exports.
For the 2nd question:
Basically, if the bitrates used in AVI and MKV are the same, the sizes of output files should be almost the same. (Strictly speaking, AVI file will be very little bit bigger than MKV due to their structural difference but it's not an issue here.) So, probably the stream bitrate on the AVI is higher than the MKV's one. However, this doesn't mean the media quality of the MKV is inferior than the AVI's one because codes can be different. For your case, probably the AVI used DIVX-like codec whilst the MKV used h.264 which provides much better video compression. But please don't misunderstand this - AVI and MKV are just containers (so is FLV) which can have (almost) anything whilst DIVX and H.264 are codecs which actually populate the data in the containers. Codecs and bitrate settings mainly contribute on the file size.
Interestingly, there is a tendency for AVI files to have video streams encoded in DIVX-like codecs and for MKVs to have ones in H.264 so many think MKVs have better video quality or smaller file size, but this is not technically true.