Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am running my crawler on sites. I was thinking that I should change my IP before running the crawler so that sites do not block me. I tried the Proxy Switcher software, it worked great.

Then I visited the http://whatismyip.com to check. But it showed my two IPs. It shows 'proxy detected'.

I would like to know what is the use of a proxy server if my original IP is still detected?

share|improve this question

1 Answer 1

up vote 0 down vote accepted

Chances are the proxy you are using sends the X-Forwarded-For header.

I was thinking that I should change my IP before running the crawler so that sites do not block me.

It sounds like you're inventing a solution to a problem that doesn't exist. If you don't want sites to block you, respect their robots.txt, don't crawl their sites every two seconds and you'll likely be fine.

As far as trying to hide behind 7 proxies etc, you might want to switching to a proxy that doesn't forward your IP (it's optional to do so), or a VPN.

I would like to know what is the use of a proxy server if my original IP is still detected?

In a corporate environment they're used for caching and access control.

share|improve this answer
    
is there any example of proxy server which don't forward my ip –  user191542 Jan 22 '13 at 1:45
    
@user191542 I believe some proxy lists have that information, whether each proxy is "anonymous" or not. –  ta.speot.is Jan 22 '13 at 6:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.