I've a very inexpensive virtual server but I get regular stuck in my developement with xdebug and Typo3 websites. Currently I'm running sshd, ftp, xdebug, mysql, postfix on my server and it's only for little developement because mostly I develop on my local box. Is xdebug requirement is heavy?
There is an overhead inevitable, though that is getting better. It still exists though, even if it's marked as disabled.
Personally, I'd leave Xdebug off a production server, and try to only run it locally. For profiling, running it live can still be useful, but XHProfLive has been engineered with less no-op overhead, and so I've had that running 1 time in 10,000 live calls before. Most people shouldn't need it anyway (except on a development machine).