0

Does anyone know how to save a web page with scripts by using wget? Or what parameters have to use to achieve?

The web page contains scripts and seems only the SingleFile Core extension in Chrome can save all the contents property. Firefox can't save the table contents and Chrome built-in save will stuck showing 'please wait' message for the result page.

The other parts on the page is not important, only the contents in the table is, so I decide to find a way to do that but spent an hour to analyze the source code and view the information from the f12 developer tools but can't a way to save the contents of the table only.

Here is the web page: http://m.marine.gov.mo/seawayScheduled/RealTimeSailing.aspx?type=dataDepartureXML&local=PE&lang=ENG

Thanks for the help!

2
  • This kind of page typically uses "AJAX": the page itself has no data and some JavaScript code sends an HTTP request to retrieve the bare data (as XML, it seems in this case), which is then parsed and inserted in the HTML. It is much more efficient to find that call and emulate it. You use a toool such as LiveHTTPHeaders (Firefox) or the built-in debugger to find the requests that are sent by the JavaScript code.
    – xenoid
    Jul 14, 2017 at 15:47
  • Thanks for the reply! There are 8 JavaScript lines and 3 unknown resource lines with the help by LiveHTTPHeaders but all of them are codes and I don't know how can they retrieve the XML file. Sorry but I'm only familiar with HTML.
    – Sam
    Jul 14, 2017 at 16:17

1 Answer 1

0

And the answer is:

wget --header 'Content-type: application/json' --header 'Cookie: ASP.NET_SessionId=1bsq4nw4nj30groly5pv5zt3' --post-data='{"local":"PE","type":"dataDepartureXML"}' 'http://m.marine.gov.mo/seawayScheduled/RealTimeSailing.aspx/GetData'

Where the 'ASP.NET....' bit is a cookie which is unfortunately mandatory. This cookie is sent when you retrieve the main page. So you have to it a bit differently with two commands:

  • one that gets the main page and discards it (but keeps the cookie in cookie.txt):

    wget --keep-session-cookies --save-cookies cookie.txt 'http://m.marine.gov.mo/seawayScheduled/RealTimeSailing.aspx?type=dataDepartureXML&local=PE&lang=ENG' -O /dev/null
    
  • one that gets the data, sending the cookie:

    wget --load-cookies cookie.txt --header 'Content-type: application/json' --post-data='{"local":"PE","type":"dataDepartureXML"}' 'http://m.marine.gov.mo/seawayScheduled/RealTimeSailing.aspx/GetData' -O data.json
    

The data you get is a JSON array with two string elements, which are raw HTML. Special characters (quotes, bracket....) are escaped so the result isn't very readable as such, bit not hard to clean up with a few global changes in an editor.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .