How to troubleshooting the connectivity issue between 2 hosts in LAN

Question

I have a home LAN (router D-LINK 524) which has a Windows XP laptop and a Mac.

The Mac is running ssh, VNC server and RDP server and listening to 22/5900/3389.

When pinging/telnet those ports from Windows machine, all got timeout. RDP to Mac from Windows fails too. The weird thing is when launch the VNC client and connect to Mac from Windows will work. After this connection established, either close it or leave it there, pinging/telnet all Mac listening ports will work.

What could cause this? The pinging suppose to work without establishing VNC connection. Is there any way/tools to troubleshoot this issue? Please advise, thanks.

EDIT:

I've tried to disable firewall/ antivirus software on both Win/Mac. But still unable to resolve the issue.

I found that either enable/disable firewall doesn't make any effect to the issue. Usually on Mac pinging Win, it may timeout for first few times then get pinging response. After Mac is able to ping Win. Win can ping Mac back. Using Wireshark on Win can see the ICMP packets and there's no ARP packets.

But in reverse way, keep pinging Mac from Win will always timeout. Using Wireshark on Win can see the ARP packets didn't get any response.

The firewall allows src:LAN.* dst:*.* protocol:*.

Finally, it seems if the ARP table on Win doesn't have Mac's info. The connection can't be established. So adding a static ARP entry on Win can resolve the issue. I wonder why the D-link 524 router ARP table can't provide Mac's info to Win.

Answer 1

I would check any firewall software running on both the Mac (including OS X's built-in firewall) and the router to see if they might be interfering with the connection. You could temporarily disable what you find to see if this solves the issue.

Another useful avenue of investigation might be the Console application, which will let you look at the Mac's logs. There might be an entry there that could elucidate what the issue is.

Answer 2

This sounds like it is your windows firewall. It isn't allowing incoming connections until it first establishes at outgoing connection to that computer. Check out your windows firewall settings for possible resolutions by either allowing certain ports, or allowing connections from local area network computers.