Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using wget for Windows (gnuwin32 wget-1.11.4-1) in Windows 8 and using it for a helpdesk tool called Kayako, telling it to poll from an email queue. The command line looks like this:

wget.exe -O null --timeout 25

I know it takes around 20 seconds to receive a response from the server in my particular case when using a browser with the URL in the command line above. However, when using that command, it returns almost immediately. This is an excerpt of the output:

Connecting to[]:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html]

I would like to know what would be the difference between the two cases and how could I get wget to behave in the same way as the browser (I know it doesn't because Kayako is not polling from the email queue).

share|improve this question
up vote 1 down vote accepted

Some websites offer browser optimized pages. That means the site checks on a request what kind of browser you are using and serves you with an optimized page or as a fallback a standard page.

Also java-script could be a problem see link

share|improve this answer
wget has a --user-agent option - try specifying a well known one. Use this site to see your current user agent string. Also, you may need to use the --referrer option - some sites don't like blank referrers. The referrer you want to send is the URL of the page containing the link you are trying to download. Also you might need to specify -e robots=off. – LawrenceC Mar 28 '13 at 3:01

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.