I have had that happen with several different .ass subtitles. For example subtitles with the
Plain with border style were not displayed correctly in these subtitles:
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,Trebuchet MS,40,&H00E6F4FC,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,1,1,2,10,10,10,1
Style: Plain with border,Arial,30,&H00000000,&H000000FF,&H00FFFFFF,&H00000000,0,0,0,0,100,100,0,0,1,2,0,5,10,10,10,1
It was fixed by changing the bold attribute from 0 to -1 or just deleting the styles completely. MPlayerX has also displayed those subtitles correctly.
You can extract subtitles from mkv files with mkvextract:
brew install mkvtoolnix
mkvextract tracks video.mkv -c UTF-8 3:subs.srt
Another common issue is that VLC shows characters that are not included in the default subtitle font as rectangles. It can be fixed by just changing the default font. You can also change the text rendering module to CoreText font renderer, but it makes text look too thin in my opinion.