Right now I have a physical server with Windows Server 2012 in which I am running a windows 7 virtual machine with Hyper-V Manager. How can I access this specific virtual machine without going through my physical server and then selecting the virtual machine from Hyper-V manager? Can this be done with port forwarding? If so, how?
A Hyper-V virtual machine shares the immediate (local) network with the Hyper-V host, unless it is connected to a private virtual switch in which case there is no reaching from the host to VM or the other way around.
If it is otherwise connected to an external or internal virtual switch, both the host and the VM use one virtual port each on that switch, which in turn connects to your very real gateway as mediated by the physical network interface. As soon as you create your first virtual switch, Windows installs virtual adapter drivers and "transparently" re-connects the Hyper-V host to the gateway through a virtualized network interface which enables virtual switching.
The thing to remember here is that even though the internal/external switch is virtual, it behaves just as your ordinary MAC switch would -- just switching Ethernet frames around based on MAC addresses. Meaning that your virtual machine appears to your gateway as if it was a physical machine, sibling to your host. So, typically your host could get assigned something like 192.168.1.10 for the IP address, while your VM would get 192.168.1.11 or something like that.
You would reach the VM by connecting with Remote Desktop Client to 192.168.1.11, no difference from how you would do so had 192.168.1.11 been a physical workstation.
What you need to check for is whether the [Windows] firewall on the VM allows for traffic on the typical RDP port -- 3389, whether Windows on the VM enables and allows RDP connections (Use Advanced System Properties dialog to check) and whether the VM is routable at all, which it typically should. The most common pitfalls is disabled RDP on the VM and/or restrictive firewall policies.
And remember -- there is nothing special about the VM as far as your local network is concerned -- it is just a machine to it, like your host.
Are you using remote desktop to connect to your VM host and running Hyper-V Manager on the VM host? You can also install Hyper-V Manager on your local machine and connect to a remote server (your VM Host machine).