Assuming this is a Windows XP SP2 or later client:
If the server uses NAT, then see Microsoft's KB 885407, "The default behavior of IPsec NAT traversal (NAT-T) is changed in Windows XP Service Pack 2".
According to Wikipedia:
NAT traversal and IPsec
In order for IPsec to work through a NAT, the following protocols need to be allowed on the firewall:
- Internet Key Exchange (IKE) - User Datagram Protocol (UDP) port 500
- Encapsulating Security Payload (ESP) - IP protocol number 50
or, in case of NAT-T:
- IPsec NAT-T - UDP port 4500
Often this is accomplished on home routers by enabling "IPsec Passthrough".
The default behavior of Windows XP SP2 was changed to no longer have NAT-T enabled by default, because of a rare and controversial security issue. This prevents most home users from using IPsec without making adjustments to their computer configuration.
It is not clear to me if machines that have been upgraded to XP SP2 are also subject to the new default. I guess they are, and if NAT is indeed used in your situation then I assume the older machine is running 2000 or earlier? Or if upgraded machines are not subject to the new default, then maybe the older machine was installed with XP prior to SP2, and then upgraded to XP SP2 or later? (Please provide some details for later readers.)
(The required mappings from the first lines of the Wikipedia quote makes it impossible to have multiple VPN clients behind the same cheap home router that uses manual port mapping. The router can then only apply that mapping towards one of the clients. But I assume most modern home routers will support multiple clients nowadays.)
ipconfig
will show you some internal IP address for some default gateway. Next, that IP address might reveal some brand/type info of the modem/router when pasting it in your browser.)