Normal performance will be good because VNC hooks in to the system and watches for new draws to controls. The latest version of VNC has a mirror driver that instead of polling for new draws is told when they occur, so it has even less work to do.
A 3D application on the other hand, draws to the entire window. If you have this application running at 640 by 480 and with 256 colours (1 byte per pixel) then you have 9mb/s to transfer every second assuming UltraVNC has no overhead (which it does). You can probably get away with this if you're on a 100mbit ethernet link (max 12.5mb/s) but once you start changing the simple parameters I gave (say you use 800 x 600) then you are in trouble.
Also note that UltraVNC applies compression to the data you are sending, so the CPU also needs to be capable of compressing 9mb/s