I do not think that this is possible using calibre. Handling images maybe somehow be possible, but since mathematical expressions are usually stored as text they will likely be screwed up by any conversion efforts.
Instead I would suggest you take a look at k2pdfopt. This allows you to generate a new PDF optimized for your reader's display size.
It can handle multiple columns, resize and reflow text, preserve images and formulas. Tables are often preserved as well, but stay small. Just try it, maybe it works for you.
If you like the results and do not mind jailbreaking your Kindle, you can run the program directly on it. I give some more details on this in this answer on StackExchange's ebook site.