Background: I have TP-Link Archer D7 router which uses a local web page for its configuration. I would like to download two given pages automatically using wget (as a daily precaution to check whether config was changed by somebody) -- for this I need direct, given, address of a page.

Some pages show you all the time the main address, despite internally they navigate among various pages. The problem is, that you cannot go to page X, because even if you are at page X, the url states you are at main page.

My case is such, I am looking at advanced status page of my router, yet, the URL still shows (main page of the router). So for example if I copy&paste this url to another tab, I will get main page, instead of status. I peeked (using firebug) what page is fetched when I switch to the status page -- However when I enter this page in URL box, I get error 403 Forbidden. It is not authentication problem because it is the same browser.

My question is -- what do I miss? What should I do, to use direct urls?

  • Can you use the tamperdata plugin or equivalent and look at what HTTP data is being sent? – user193661 Sep 22 '15 at 12:42
  • @user193661, sure, firebug can show HTTP traffic as well. What I should look for? As I already wrote, I found out single GET for the address which interests me, but when I copy&paste it I get error 403. – greenoldman Sep 22 '15 at 13:21
  • 1
    You need to "replay" the request in the browser, so the correct headers are used. – user193661 Sep 22 '15 at 13:27
  • @user193661, thank you. With help of this post superuser.com/questions/584918/… and then Header Hacker from Chrome I was able to get the direct page (this TP-Link stuff is ordeal...). The problem was with missing Referer header when accessing the page directly. Could you please post your comment as an answer so I could accept it? – greenoldman Sep 22 '15 at 16:24

If you unexpectedly get 403 HTTP response, the web server may be checking for certain headers. So you should check that you are using the correct HTTP headers. You can do this with a web browser plugin like Tamper Data for Firefox or Header Hacker for Google Chrome.

Example HTTP GET request and response:

  5:40:08.691[72ms][total 72ms] Status: 302[Moved Temporarily]
  GET Load Flags[LOAD_DOCUMENT_URI  LOAD_INITIAL_DOCUMENT_URI  ] Content Size[-1] Mime Type[text/html]
     Request Headers:
        Accept-Encoding[gzip, deflate]
     Response Headers:
        Expires[Tue, 22 Sep 2015 13:10:12 GMT]
        Date[Tue, 22 Sep 2015 12:40:12 GMT]
        Last-Modified[Tue, 22 Sep 2015 12:40:12 GMT]

Option 1, Hover over links

In all modern browsers you can hover over the link and it will show you the page URL, like this: cPanel 1 cPanel 2 and harvest the URL that way

Option 2, Inspect element

You could inspect element like so: Inspect Element 1 Inspect Element 2 and copy that URL but add at the start

Option 3, Show only this frame

As far as I know, this only works in Firefox. Since these are iframes, you can right click in the frame and select Open Frame in New Tab and harvest the URL as below: View frame

  • He already knows what page will be fetched "I peeked (using firebug) what page is fetched when I switch to the status page -- However when I enter this page in URL box, I get error 403 Forbidden." – DavidPostill Sep 22 '15 at 11:32
  • @DavidPostill I've seen this before where the status.htm is fetching something else entirely – td512 Sep 22 '15 at 11:33
  • Thank you, but (1) I don't have direct URL given somehow in the main page, this what I would like to read; in this case it is javascript:void(0), (2) same as (1), (3) it does not have to be iframe, modern, dynamic pages go beyond frames, and unfortunately it is my case. – greenoldman Sep 22 '15 at 11:43

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.