So I just built a computer and installed the Nvidia Geforce GTX 660. I already connect my computer to the monitor via VGA, but the install manual says that I have to connect my graphics card to my monitor via VGA. Can I just connect it by HDMI or what?
Your motherboard has an integrated VGA port. If you connect your monitor to that port, you are not using your NVIDIA card but the MB's on-board one,
In order to use your card, you need to connect your monitor to a port on your card. Since your card has no VGA port, you need to connect your monitor to it either via HDMI or DisplayPort or DVI.
If you are connected via a VGA port, then you are not connected to your graphics card but just to your integrated graphics chip.
According to the GeForce GTX-660Ti, you can connect directly to the HDMI. Page 11 addresses this.
Connect your computer monitor(s) to the GeForce graphics card using the connectors shown. A single GeForce GTX 660 Ti graphics card can support up to four displays in a multi-monitor configuration.
As you can see, there is no VGA port, so you must connect to another port. Once you've installed the hardware, you have to install the drivers per the instructions.
Page 21 covers configuring HDMI.