Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I’m trying to figure out a way to use WGET or a similar tool so that I can schedule a web page to be downloaded regularly as a sort of updating log. The problem is that the page requires that I be logged in otherwise I get a different page, generic. Further, the page does not take login information as GET parameters in the URL, it uses POST to log in on the login page and cookies to save the login information that’s read by the regular page.

I’m currently using GNU Wget 1.10.2 for Windows. I’ve tried using WGET’s cookie functionality but have had mixed results, usually skewing towards it not working.

Can anyone please advise on a way to accomplish this? Thanks a lot.

share|improve this question
    
I've heard that curl and PHP can do it but am not sure how –  barlop Feb 21 '11 at 20:24
    
See superuser.com/questions/257831/…. –  Ben Alpert Mar 30 '11 at 5:05
add comment

2 Answers

It sounds like you want some kind of web automation tool rather a straight downloader like wget.

The one that comes to my mind is WatiN but there are many others like this.


Edit: Actually, Selenium is probably a better fit. If you're not a programmer, it has a simple point and click "macro" type mode in Firefox.

share|improve this answer
    
Those don’t look like what I am looking for. I need a way of running a program from a script or something so that it can be scheduled to run automatically in the background now and then. –  Synetech Feb 21 '11 at 21:35
add comment

Does the page have a "Remember me" option. If so, you can export the cookie file (see this: http://blog.mithis.net/archives/python/90-firefox3-cookies-in-python) and use --load-cookies in wget.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.