I have a ping to 127.0.0.1 which varies between 1 and 23 milliseconds. I'm using a VIA Rhine II Fast Ethernet Adapter, and updating to the latest driver version 184.108.40.206 didn't seem to fix the problem.
It sounds like you might have a service running locally that might be producing this behavior. That is the only thing I have ever seen that would cause a response from localhost to exceed <1ms.
take a look at any connections that are ESTABLISHED or LISTENING.
Some common services to look for would be HTTP (80), Socks Proxy (8080, 8000, etc), SQL Server (1433), Webmin (10000), MySQL (3306).