I have a mail application running on a Windows 7 machine. I need to allow a reverse lookup, as of right now when a reverse lookup is ran it timesout. Is there a port that needs to be open or software to be installed?
There is nothing you need to allow specifically for reverse lookup (it is just normal DNS via UDP 53), but most likely you are facing the problem that your mail server is not accepted by other mail servers because of lacking reverse DNS. Especially, this has nothing to do with your internal DNS, but rather with the "official" internet DNS. It is sometimes confusing for beginners whom to ask for remedy, but it is logical after all:
When you buy a domain example.com at some registrar, you also register a few DNS servers that are authoritative for the zone example.com (or possibly the registrars servers are used by default). So to make changes to the example.com zone you have to modify the data at these name servers (or have your registrar change it).
The reverse DNS is not related to example.com, so not handled by the registrar. Instead, the corresponding sub-zone of in-addr.arpa is delegated to the one providing you with ip addresses, that is your access provider. You'll have to contact them to configure the corresponding rDNS entries (or, in case you have a rather large subnet, have the zone delegated to name servers under your immediate control, but you won't want that).
You need to put a reverse lookup record into your DNS server so that your mail server's FQDN name can be found by IP.
More info: Reverse DNS lookup