I'm looking for a program to spider a web site* and download the content to disk. I've got the following requirements though which appear to trip up the programs I've tried:
- The site's logon process is distributed between multiple pages so I need to be able to start the session manually with my web browser, authenticate, then launch the spider and give it access to the cookies.
- Many of the links execute Javascript and then return content.
- Many of the links return PDF documents (some spiders do support this).
So far though I'm unable to find a spider that can handle #1 and #2.
* The site is a bill payment service and it has my bill history. I'm looking to stop using the service but I want to keep the history.
