Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When I visited this website throught my Firefox 13, I got a page with some content. But when I use wget to download it :


I got other content on downloaded HTML page. Tried set user-agent :

wget -U 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1'

but got same result.

What happened ? and how do I get the same result as when I visit it throught Firefox?

Thank you in advance for any help you can provide.


Here is from Firefox => view source:

<!DOCTYPE html>

<html dir="ltr" lang="vi">  

    <head id="ctl00_page_header">            
                Tinhvan Group - Trang chủ

and here from downloaded by wget

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="">
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><link href="Content/images/main.css" rel="stylesheet" type="text/css" /><link href="Content/images/mail-detail.css" rel="stylesheet" type="text/css" />
    <script src="../../Content/JqueryUI/js/jquery-1.3.2.min.js" type="text/javascript"></script>    

    Trang chủ - Tinhvan Group Website
share|improve this question
What "other content"? – user3463 Jul 31 '12 at 8:58
@RandolphWest : I've updated the question – HVNSweeting Jul 31 '12 at 9:19
Send the exact same header and cookies in your wget request (the result may depend not just on user-agent). Also disable javascript in your browser. Look for meta refresh's. – artistoex Jul 31 '12 at 9:31
I have actually tested it with the same headers as in Firefox and it is not working.. – mnmnc Jul 31 '12 at 10:13

Try to use CURL

curl --location

It will go to the redirected page if it provided.

You can also try

curl -I
curl -I --location

and paste in here so we will know what headers are returned.

I did checked:

wget -S --header "User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header "Accept-Encoding: gzip,deflate,sdch" --header "Accept-Language: en-US,en;q=0.8" --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3"

And no luck yet..

HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Cache-Control: private
  Content-Type: text/html; charset=utf-8
  Expires: Tue, 31 Jul 2012 10:08:38 GMT
  Server: Microsoft-IIS/7.0
  Set-Cookie: .ASPXANONYMOUS=0EsBS5alzQEkAAAAZjllMGZmMDMtMTQ3Zi00ZDZlLWJiYWUtZGJhNDlhZjY1NWZmhWyvGlE9QG7-yO6wBOECE3K4rkA1; expires=Mon, 08-Oct-2012 20:48:38 GMT; path=/; HttpOnly
  X-AspNetMvc-Version: 1.0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  Date: Tue, 31 Jul 2012 10:08:38 GMT
  Connection: keep-alive
  Content-Length: 9874
Length: 9874 (9.6K) [text/html]

It is possible that js which is loading is performing a silent redirection of some kind which is changing the page source. wget / curl does not have javascript suppor - that is why content is different.

TCPDUMP record of wget download:

To long for direct paste so i placed it here: click

For further analysis.

share|improve this answer – HVNSweeting Jul 31 '12 at 9:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .