I currently have a computer running a GTX 560 Ti with two monitors (one in each of the DVI ports). Is it possible to add a third display (without any extra hardware or peripherals) by plugging it into the mini HDMI port? What if I plugged the third monitor into any of the onboard graphic ports, what would happen?
No you can't.
According to NVIDIA's specs:-
NVIDIA GeForce GTX 560 Ti, Graphics Card Installation Guide. (see page 10), the GTX560Ti only supports 2 monitors. You can't even use 2 DVI and the mini HDMI port simultaneously because of the card internal wires layout, they use the same connection.
Try plugging the third monitor into the onboard graphic port if you have one, if you don't you'll need another graphic card, or a card wich supports 3 monitors like a AMD >HD6000 or a Geforce >GTX600.