1

When I visited this website throught my Firefox 13, I got a page with some content. But when I use wget to download it :

wget http://tinhvan.com

I got other content on downloaded HTML page. Tried set user-agent :

wget -U 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1' http://tinhvan.com

but got same result.

What happened ? and how do I get the same result as when I visit it throught Firefox?

Thank you in advance for any help you can provide.

UPDATE

Here is from Firefox => view source:

<!DOCTYPE html>

<html dir="ltr" lang="vi">  

    <head id="ctl00_page_header">            
            <title>    
                Tinhvan Group - Trang chủ

and here from downloaded by wget

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><link href="Content/images/main.css" rel="stylesheet" type="text/css" /><link href="Content/images/mail-detail.css" rel="stylesheet" type="text/css" />
    <script src="../../Content/JqueryUI/js/jquery-1.3.2.min.js" type="text/javascript"></script>    
    <title>

    Trang chủ - Tinhvan Group Website
4
  • 2
    What "other content"?
    – user3463
    Jul 31, 2012 at 8:58
  • @RandolphWest : I've updated the question Jul 31, 2012 at 9:19
  • Send the exact same header and cookies in your wget request (the result may depend not just on user-agent). Also disable javascript in your browser. Look for meta refresh's.
    – wnrph
    Jul 31, 2012 at 9:31
  • I have actually tested it with the same headers as in Firefox and it is not working..
    – mnmnc
    Jul 31, 2012 at 10:13

1 Answer 1

0

Try to use CURL

curl www.webpage.com --location

It will go to the redirected page if it provided.

You can also try

curl -I www.webpage.com
curl -I --location www.webpage.com

and paste in here so we will know what headers are returned.

I did checked:

wget -S --header "User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header "Accept-Encoding: gzip,deflate,sdch" --header "Accept-Language: en-US,en;q=0.8" --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3" tinhvan.com

And no luck yet..

HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Cache-Control: private
  Content-Type: text/html; charset=utf-8
  Expires: Tue, 31 Jul 2012 10:08:38 GMT
  Server: Microsoft-IIS/7.0
  Set-Cookie: .ASPXANONYMOUS=0EsBS5alzQEkAAAAZjllMGZmMDMtMTQ3Zi00ZDZlLWJiYWUtZGJhNDlhZjY1NWZmhWyvGlE9QG7-yO6wBOECE3K4rkA1; expires=Mon, 08-Oct-2012 20:48:38 GMT; path=/; HttpOnly
  X-AspNetMvc-Version: 1.0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  Date: Tue, 31 Jul 2012 10:08:38 GMT
  Connection: keep-alive
  Content-Length: 9874
Length: 9874 (9.6K) [text/html]

It is possible that js which is loading is performing a silent redirection of some kind which is changing the page source. wget / curl does not have javascript suppor - that is why content is different.

TCPDUMP record of wget download:

To long for direct paste so i placed it here: click

For further analysis.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .