No OST files in Outlook for Mac. The mails are stored in /Users//Documents/Microsoft User Data/Office 2011 Identities/
:Messages paulhargreaves$ pwd
/Users/paulhargreaves/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/Data Records/Messages
:Messages paulhargreaves$ find . | wc -l
I have lots of mails over 10MB in my mailbox so it is unlikely to be some "10MB" limit. It's becoming difficult to find powerpoint documents smaller than 10MB these days.
I suspect the problem is elsewhere, unfortunately. Outlook for Mac can tend to crash but you can perform a recovery by holding the alt key when re-launching the app to perform a rebuild of the database.
Also check in Console to see what errors may be reported at the time Outlook is crashing. Sometimes, if you are lucky, it may give you something you can use to diagnose further.