I'm looking for an application/tool that will allow me to spider a website where you have to scroll down vertically repeatedly for the whole page to load. I need to be able to define the starting URL, and define rules to require or disallow character patterns as to what URLs are saved to a file.
This task could be done with a VBScript (which runs natively under Windows) as is seen here: http://www.example-code.com/vbscript/spider.asp - but only for pages that completely load without the user scrolling down.
The application/tool needs to be freely available. (For example, VBScript runs native to Windows, you just save a text file as .vbs and execute it.)
The application/tool also needs to be able to log in to the website (https://) with a user name and password.