A yellow exclaimation mark in the connection icon means that one of the connected networks does not give you internet access.
If you have more than one network connections and only one of them has no internet connection, the icon already pops up. For example, if you have a wifi and LAN connection to the same network, but one is not giving you internet access. A VPN connection could also give this if it is configured to act as a gateway (by default it does)
Windows XP does not have such icon so it is logical that it doesn't show up there.
Alternatively, there are cases known where internet was briefly disconnected and the icon showed but didn't disappeared even though things corrected itself. Usually a restart fixes those issues. If this keeps happening, troubleshoot the network and look into a faulty router config or network cable. Also make sure people are not connecting to both LAN and Wifi at the same time, because one often cancels the other.