Generally what controls the machine name resolution is the DNS server, which in this case is probably going to be the router connected directly to your modem (unless your modem is a modem/router combo).
The DNS is what finds the ip for the computer names, but the second router is probably trying to act as the DNS as well. So think of it this way, Router 1 can find all machine addresses because all computers go through it for the network/internet, but Router 2 can only check the computers connecting through it. I would check the DNS setting on Router 1, and put the same settings into Router 2 (should be the IP for your Router 1 or Modem).
You want to use the same DNS on the entire network, or you run into issues like that where your network is split into different pieces.
I hope that makes sense, if not I'll try to clarify whatever I didn't word correctly.