If all of your computers are running Windows Vista or higher, you should be able to refer to them by name already. If you aren't sure about the names just click Start->Network in Vista/Win7.
If you have a mixture of operating systems, you will need to either edit the local hosts files on each machine (\windows\system32\drivers\etc\hosts for windows machines, /etc/hosts for others), OR setup a DNS server on your network and set your local DNS server as the default DNS in your router. And old computer (even REALLY old like a pentium 60) running linux can do this very nicely. There are also DNS servers for windows, but I am not personally familiar with any of them.
EDIT: For completeness, you should also consider installing an alternate OS on your router if it supports it. I'm running Tomato but DD-WRT is probably the most popular. See this Wikipedia page for more info:
http://en.wikipedia.org/wiki/List_of_wireless_router_firmware_projects
Thanks to leif81 for mentioning this first.