When I visited this website throught my Firefox 13, I got a page with some content. But when I use wget to download it :

wget http://tinhvan.com

I got other content on downloaded HTML page. Tried set user-agent :

wget -U 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1' http://tinhvan.com

but got same result.

What happened ? and how do I get the same result as when I visit it throught Firefox?

Thank you in advance for any help you can provide.


Here is from Firefox => view source:

<!DOCTYPE html>

<html dir="ltr" lang="vi">  

    <head id="ctl00_page_header">            
                Tinhvan Group - Trang chủ

and here from downloaded by wget

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><link href="Content/images/main.css" rel="stylesheet" type="text/css" /><link href="Content/images/mail-detail.css" rel="stylesheet" type="text/css" />
    <script src="../../Content/JqueryUI/js/jquery-1.3.2.min.js" type="text/javascript"></script>    

    Trang chủ - Tinhvan Group Website
  • 2
    What "other content"? – user3463 Jul 31 '12 at 8:58
  • @RandolphWest : I've updated the question – HVNSweeting Jul 31 '12 at 9:19
  • Send the exact same header and cookies in your wget request (the result may depend not just on user-agent). Also disable javascript in your browser. Look for meta refresh's. – artistoex Jul 31 '12 at 9:31
  • I have actually tested it with the same headers as in Firefox and it is not working.. – mnmnc Jul 31 '12 at 10:13

Try to use CURL

curl www.webpage.com --location

It will go to the redirected page if it provided.

You can also try

curl -I www.webpage.com
curl -I --location www.webpage.com

and paste in here so we will know what headers are returned.

I did checked:

wget -S --header "User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header "Accept-Encoding: gzip,deflate,sdch" --header "Accept-Language: en-US,en;q=0.8" --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3" tinhvan.com

And no luck yet..

HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Cache-Control: private
  Content-Type: text/html; charset=utf-8
  Expires: Tue, 31 Jul 2012 10:08:38 GMT
  Server: Microsoft-IIS/7.0
  Set-Cookie: .ASPXANONYMOUS=0EsBS5alzQEkAAAAZjllMGZmMDMtMTQ3Zi00ZDZlLWJiYWUtZGJhNDlhZjY1NWZmhWyvGlE9QG7-yO6wBOECE3K4rkA1; expires=Mon, 08-Oct-2012 20:48:38 GMT; path=/; HttpOnly
  X-AspNetMvc-Version: 1.0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  Date: Tue, 31 Jul 2012 10:08:38 GMT
  Connection: keep-alive
  Content-Length: 9874
Length: 9874 (9.6K) [text/html]

It is possible that js which is loading is performing a silent redirection of some kind which is changing the page source. wget / curl does not have javascript suppor - that is why content is different.

TCPDUMP record of wget download:

To long for direct paste so i placed it here: click

For further analysis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.